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BAKER & BOTTS, L.L.P. 
30 ROCKEFELLER PLAZA 
NEW YORK, NEW YORK 10112 



TO ALL WHOM IT MAY CONCERN: 

Be it Imown that WE, SHREE K. NAYAR, a citizen of the United States residing 
in County of New York, State of New York, and TOMOO MITSUNAGA, a citizen of Japan, 
residing in Kawasaki City, Japan, whose post office addresses are , respectively, 560 Riverside 
Drive, New York, NY 10027, aad 1-9-3-403, Hisamoto, Takatsu, Kawasaki City, Kanagawa 
Prefecture, 213-001 1, Japan, have invented an improvement in 

METHOD AND APPARATUS FOR OBTAINING HIGH DYNAMIC RANGE IMAGES 
of which the following is a 

SPECIFICATION 
BACKGROUND OF INVENTION 

The present invention relates to method and apparatus for capturing an image of a scene, 
and, more particularly, to method and apparatus for capturing a relatively high dynamic range 
image using a relatively low dynamic range image sensor. 

Virtually any real world scene produces a very large range of brightness values. By 
contrast, known image sensing devices have very limited dynamic ranges. For example, it is 
typical for a video image sensor to provide only 8-bits or less of grey-scale or color information. 
In the case of grey-scale images, 8-bits allows only 256 discrete grey-scale levels, which is not 
sufficient to capture the fine details of most real hfe scenes. 
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A known solution to the problem of capturing high dynamic range images with a low 
dynamic range unage sensor is to capture multiple images of each local area of the scene while 
varying the exposure to light from the scene. There are three general variations of this solution: 
(1) temporal variation of the exposure while capturing a sequence of images, (2) using multiple 
image sensors having different sensitivities to light from the scene and (3) spatial variation of the 
exposure. 

Typically, temporal exposure variation involves sequentially capturing multiple images 
of the scene with different exposures and then combining the multiple captured images into a 
single high dynamic range image. The use of temporal exposure variation is disclosed, for 
example, in: Japanese Patent No. 08-331461 entitied "Driving Metiiod for Solid-State Image 
Pickup Device" to Hamazaki, issued December 1996; Japanese Patent No. 07-254965 entitled 
"Electronic Image Pickup Device" to Saito, issued February 1995; U.S. Patent No. 5,420,635 
entitied "Video Camera, Imaging Method Using Video Camera, Metiiod of Operating Video 
Camera, Image Processing Apparatiis and Metiiod, and Solid-State Electronic Imaging Device" 
to Konishi et al, issued May 1995; U.S. Patent No. 5,455,621 entitied "Image Method for a 
Wide Dynamic Range and an Imaging Device for a Wide Dynamic Range" to Morimura, issued 
October 1993; Japanese Patent No. 06-141229 entitied "Image Picking Up and Synthesizing 
Metiiod and Image Pickup Device witii High Dynamic Range" to Morimura, issued May 1994; 
U.S. Patent No. 5,801,773 entitied "Image Data Processing Apparatus for Processing Combined 
Image Signals in Order to Extend Dynamic Range" to Ikeda, issued September 1998; U.S. Patent 
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No. 5,638,1 18 entitled "Image Device with Diverse Storage Times Used in Image Composition" 
to Takahashi et aL issued Jime 1997; U.S. Patent No. 5,309,243 entitled "Method and Apparatus 
for Extending the Dynamic Range of an Electronic Imaging System" to Tsai issued May 1994; 
Mann and Picard, "Being 'Undigital' with Digital Cameras: Extending Dynamic Range by 
Combining Differently Exposed Pictures," Proceedings of ISTs 48th Annual Conference, pp. 
422-428, May 1995; Debevec and Malik, "Recording High Dynamic Range Radiance Maps for 
Photographs," Proceedings of the ACM SIGGRAPH, 1997, pp. 369-378, August 1997; and 
Mitsunaga and Nayar, "Radiometric Self Calibration," IEEE Conference on Computer Vision 
and Pattern Recognition (CVPR 99), pp. 374-380, June 1999. However, known temporal 
exposure variation techniques all have the fundamental problem in that changes in the scene may 
take place during the sequential image capture at different exposures. Consequently, these 
techniques are useful only for static scenes where the radiance from the scene remain constant 
and where the position and orientation of the image sensor, and its geometric characteristics 
remain unchanged during sequential image capture at different exposures. 

Another known technique for capturing multiple images of a scene for high dynamic 
range imaging uses multiple optically aligned image sensors and a beam splitter for providing 
incident light from the same view of the scene to the multiple image sensors. The image sensors 
having different sensitivities so that multiple images are simultaneously captured at different 
exposures. The multiple images captured by the image sensors are then combined to form a 
single high dynamic range image. Such techniques, for example, are described in: Japanese 
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Patent No. 08-22341 entitled "Image Sensor" to Doi et al., issued August 1986; Japanese Patent 
No. 07-25495 entitled "Electronic Image Pickup Device" to Saito, issued February 1995; 
Japanese Patent No. 07-254966 entitled "Electronic Image Pickup Device" to Saito, issued 
October 1995; Japanese Patent No. 08-340486 entitled "Electronic Image Pickup Device to 
Saito, issued December 1996; Japanese Patent No. 10-06901 1 entitled "Image Pickup Device" to 
Kimura, issued March 1998; and U.S. Patent No. 5,801,773 entitled "Image Data Processing 
Apparatus for Processing Combined Image Signals in Order to Extend Dynamic Range" To 
Ikeda, issued September 1998. The multiple image sensor techniques have the drawback in that 
they require high precision optics and a plurality of image sensors so that it is difficult to build 
compact and inexpensive imaging systems using these techniques. 

Known spatial exposure variation techniques for high dynamic range imaging typically 
use a special image sensor having more than one (typically two) light-sensing elements having 
different sensitivities for each pixel. The light-sensing elements for each pixel are 
simultaneously exposed and their respective output signals are combined to derive a brightness 
value for the pixel. Such techniques are described, for example, in: U.S. Patent No. 5,789,737 
entitled "High Dynamic Range Segmented Pixel Sensor Array" to Street et al., issued August 
1998; U.S. Patent No. 4,623,928 entitled "High Dynamic Range CCD Detector/Imager" to 
Handy, issued November 1986; U.S. Patent No. 4,873,561 entitled "High Dynamic Range 
Charge Coupled Device" to Wen, issued October 1989; Japanese Patent No. 08-331461 entitled 
"Driving Method for Solid-State Image Pickup Device" to Hamazaki, issued December 1996; 
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Japanese Patent No. 59-217358 entitled "Charge Coupling Image Pickup Device" to Murakoshi, 
issued December 1994; and U.S. Patent No. 5,420,635 entitled "Video Camera Imaging Method 
Using Video Camera, Method of Operating Video Camera, Image Processing Apparatus and 
Method, and Solid-State Electronic Imaging Device" to Konishi et al, issued May 1995. 
Hov^ever, knovm spatial exposure variation techniques for high dynamic range imaging have the 
drawback of providing lower spatial resolution (typically a factor of two lower in the vertical 
direction) than the resolution of ordinary image sensors having the same number of light-sensing 
elements. 

Accordingly, there exists a need for a method and apparatus for capturing a relatively 
high dynamic range images using a relatively low dynamic range image sensor, which 
overcomes the problems of the prior art as discussed above. 

SUMMARY OF THE INVENTION 

In accordance with one aspect of the present invention, there is provided a method for 
capturing a relatively high dynamic range image of a scene using a relatively low dynamic range 
image sensor exposed to incident light from the scene. The image sensor has a multipUcity of 
light-sensing elements in a linear or two-dimensional array, and each light-sensing element has a 
respective one of a plurality of sensitivity levels to incident light in accordance with a 
predetermined sensitivity pattern for the array of light-sensing elements. Each of the light- 
sensing elements in the array in response to incident light from the scene produces a captured 
image brightness value at a corresponding one of a multiplicity of pixel positions of a pixel 
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position array, which may be a linear array having regularly spaced pixel positions in a row or a 
rectangular array having pixel positions at respective intersections of regularly spaced orthogonal 
pixel rows and pixel columns. In this manner each pixel position corresponds to a respective one 
of the pliirality of sensitivity levels. The method comprises the step of estimating respective off- 
grid brightness values at a multiplicity of off-grid positions from the captured image brightness 
values at the multiplicity of pixel positions of the pixel position array, the off-grid positions 
being located in a regular array at respective interstices of the pixel position array. The method 
further comprises the step of deriving pixel brightness values of a relatively high dynamic range 
image of the scene from the estimated off-grid brightness values at the multiplicity of off-grid 
positions. Where the pixel position array is a linear array in which the brightness value at each 
pixel position is captured by a corresponding light-sensing element of a linear image sensor, the 
off-grid positions are in a regular linear array, and the relatively high dynamic range image is a 
linear image represented by a regular linear array of pixel brightness values. Where the pixel 
position array is defined by respective intersections of regularly spaced orthogonal pixel rows 
and pixel columns, the brightness values of the pixel positions in the array may be captured by 
corresponding light-sensing elements of a two-dimensional image sensor or may be a 
concatenation of the brightness values at pixel positions of linear pixel position arrays 
representing line images of a scanned scene successively captured by a linear image sensor. 

In accordance with an exemplary embodiment of the method of the present invention, the 
off-grid positions are in a regular linear or rectangular array and the step of deriving pixel 
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brightness values of a relatively high dynamic range image comprises using the estimated off- 
grid brightness values as the pixel brightness values of the relatively high dynamic range image. 

According to another exemplary embodiment of the method of the present invention, the 
step of deriving pixel brightness values of the relatively high dynamic range image comprises 
resampling the estimated off-grid brightness values at the multiplicity of off-grid positions to 
derive respective resampled brightness values at the multiplicity of pixel positions of the pixel 
position array to thereby obtain the pixel brightness values of the relatively high dynamic range 
image. 

In accordance with still another exemplary embodiment of the method of present 
invention, the step of estimating respective off-grid brightness values at the multiplicity of off- 
grid positions from the captured image brightness values at the multiplicity of pixel positions 
comprises the steps of estimating a sensitivity level off-grid brightness value for each individual 
one of the pluraUty of sensitivity levels at each one of the off-grid positions using only the 
captured image brightness values at pixel positions corresponding to the individual one of the 
sensitivity levels. The sensitivity level off-grid brightness value for each individual one of the 
sensitivity levels at each one of the off-grid positions is multiplied by a predetermined weighting 
factor for the individual one of the sensitivity levels to derive respective weighted sensitivity 
level off-grid brightness values for the different ones of the sensitivity levels at each one of the 
off-grid positions. The respective weighted sensitivity level off-grid brightness values for the 
different ones of the sensitivity levels at each one of the off-grid positions are added together to 
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derive a weighted siim sensitivity level brightness value at each one of the off-grid positions. 
The weighted sum sensitivity level brightness value at each one of the off-grid positions is 
compensated by the inverse of a combined response function of the light-sensing elements to 
derive the respective estimated off-grid brightness values at the off-grid positions. The combined 
response function of the light-sensing elements is the sum of weighted response functions of 
light-sensing elements having different ones of the sensitivity levels, each weighted response 
function being a response function of a light-sensing element having a respective one of the 
sensitivity levels multiplied by a predetermined weighting factor for the respective one of the 
sensitivity levels. 

According to yet another exemplary embodiment of the method of the present invention, 
the step of estimating respective off-grid brightness values at the multiplicity of off-grid 
positions from the captured image brightness values at the multiplicity of pixel positions 
comprises estimating an off-grid brightness value at each individual one of the off-grid positions 
by first compensating the captured image brightness value at each individual one of a 
predetermined group of pixel positions that are nearest neighbors to the individual one of the off- 
grid positions by the inverse of the response function of a light-sensing element having a 
sensitivity level corresponding to the individual one of the nearest neighbor pixel positions. The 
compensated captured image brightness values at the nearest neighbor pixel position are then 
each compared with a predetermined low threshold value and a predetermined high threshold 
value, and a sum is taken of the compensated nearest neighbor pixel position brightness values 
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that are greater than the predetermined low threshold value and less than the predetermined high 
threshold value. The sum is then divided by the number of compensated nearest neighbor pixel 
position brightness values included in the sum to derive an estimated off-grid brightness value 
for the individual one of the off-grid positions. 

According to a further exemplary embodiment of the method of the present invention, 
interpolated on-grid brightness values at respective pixel positions of the pixel position array are 
derived directly from the captured image brightness values using an on-grid interpolation filter. 
The interpolated on-grid brightness value at each individual one of the pixel positions is derived 
by computing the product of the captured image brightness values at a predetermined group of 
pixel positions that are nearest neighbors to the individual one of the pixel positions and an on- 
grid interpolation filter kernel, the predetermined group of pixel positions having the same 
dimensions as the filter kernel. The interpolated on-grid brightness values are then each 
compensated by the inverse of a combined response function of light-sensing elements having 
different sensitivity levels to derive the pixel brightness values of the relatively high dynamic 
range output image. 

In accordance with another aspect of the present invention, there is provided a system for 
capturing a relatively high dynamic range image of a scene using a relatively low dynamic range 
image sensor adapted to be exposed to incident light from the scene. The system comprises an 
image sensor having a multiplicity of light-sensing elements in a linear or two-dimensional array. 
Each light-sensing element in the array having a respective one of a plurality of sensitivity levels 
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to incident light in accordance with a predetermined sensitivity pattern for the array of light- 
sensing elements. Each light-sensing element being responsive to incident light from the scene 
for producing a captured image brightness value at a corresponding one of a multiplicity of pixel 
positions of a pixel position array, which may be a linear array having regularly spaced pixel 
positions in a row or a rectangular array having pixel positions located at respective intersections 
of regularly spaced orthogonal pixel rows and pixel columns. In this manner, each pixel position 
corresponds to a respective one of the plurality of sensitivity levels of the light-sensing elements. 
The system further comprises a captured image memory for storing the respective captured 
image brightness values at the multiplicity of pixel positions produced by the light-sensing 
elements, and an off-grid estimator for deriving from the captured image brightness values in the 
captured image memory respective estimated off-grid brightness values at a multiplicity of off- 
grid positions located in a regular array at respective interstices of the pixel position array. The 
system further includes an output image generator for deriving pixel brightness values of a 
relatively high-dynamic range output image from the estimated off-grid brightness values, and an 
output image memory for storing the pixel brightness values of the output image. Where the 
pixel position array is a linear array and the captured image memory stores brightness values 
captured by corresponding light-sensing elements of a linear image sensor, the off-grid positions 
are in a regular linear array and the output image is a linear image represented by a regular array 
of pixel brightness values. Where the pixel position array is a rectangular array defined by 
respective intersections of regularly spaced orthogonal pixel rows and pixel columns, the 
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captured image memory may store brightness values captured by conesponding light-sensing 
elements of a two-dimensional image sensor or a concatenation of Unear pixel position array 
brightness values of line images of a scanned scene successively captured by a linear image 
sensor. In the latter instance, the off-grid positions are in a regular two-dimensional array and 
the output image is a two-dimensional image represented by a regular two-dimensional array of 
pixel brightness values. 

According to an exemplary embodiment of the system of the present invention, the output 
image generator provides the estimated off-grid brightness values derived by the off-grid 
estimator to the output image memory as the pixel brightness values of the output image. 

According to another exemplary embodiment of the system of the present invention, the 
system includes an off-grid brightness value memory for storing the respective estimated off-grid 
brightness values derived by the off-grid estimator at the off-grid positions, and an on-grid 
resampler for deriving from the estimated off-grid brightness values in the off-grid brightness 
value memory respective resampled brightness values at the pixel positions of the pixel position 
array. The resampled brightness values are then provided to the output image memory for 
storage therein as the pixel brightness values of the output image. 

According to still another exemplary embodiment of the system of the present invention, 
the off-grid estimator include an off-grid position generator for providing the multiplicity of off- 
grid positions and a sensitivity pattern memory for storing the respective sensitivity levels 
corresponding to the pixel positions. The off-grid estimator also includes a plurality of 
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sensitivity level off-grid brightness value estimators, each corresponding to a respective one of 
the plurality of sensitivity levels of the light-sensing elements, for receiving off-grid positions 
from the off-grid position generator and sensitivity level data from the sensitivity pattern 
memory, and for deriving from the captured image brightness values in the captured image 
5 memory a plurality of sensitivity level off-grid brightness values each corresponding to a 

different one of the plurality of sensitivity levels at each one of the off-grid positions received 
from the off-grid position generator. The off-grid estimator also includes an accumulator for 
receiving the sensitivity level off-grid brightness values derived by the plurality of sensitivity 
Lfl level off-grid brightness value estimators at each one of the off^grid positions. The accumulator 
comprises a weighting factor memory for storing respective ones of a plurality of predetermined 
weighting factors each corresponding to a respective one of the plurality of sensitivity levels, and 
fj a plurality of multipliers each for computing the product of the sensitivity level off-grid 
f y brightness value corresponding to a respective one of the plurality of sensitivity levels and the 
U weighting factor corresponding to the same sensitivity level to provide a respective weighted 
15 sensitivity level off-grid brightness value corresponding to the respective one of the sensitivity 
levels at each one of the off-grid positions received from the off-grid position generator. The 
accumulator further includes an adder for summing the weighted sensitivity level off-grid 
brightness values corresponding to respective ones of the sensitivity levels at each one of the 
received off-grid positions computed by the plurality of multipliers to derive a respective 
20 weighted sum sensitivity level off-grid brightness value at each one of the received off-grid 
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positions. The off-grid estimator further comprises a response function compensator for 
compensating each one of the weighted sum sensitivity level off-grid brightness values by the 
inverse of a combined response function of light-sensing elements to derive the respective 
estimated off-grid brightness values at the off-grid positions. The combined response function of 
5 the light-sensing elements is the sum of weighted response functions of light-sensing elements 
having different ones of the sensitivity levels, each one of the weighted response functions being 
a response function of a light-sensing element having a respective one of the plurality of 

^ sensitivity levels multiplied by a predetermined weighting factor corresponding to the respective 

,n one of the plurality of sensitivity levels. 

tp In yet another exemplary embodiment of the system of the present invention, the off-grid 

^Jr~ estimator comprises a pixel position generator for providing the multiplicity of pixel positions, a 
;3 sensitivity pattern memory for storing respective sensitivity levels corresponding to the 
U multiplicity of pixel positions, and a response function compensator coupled to receive pixel 

positions from the pixel position generator and corresponding sensitivity levels from the 
15 sensitivity pattern memory. The response function compensator also receives captured image 

brightness values from the captured image memory at the pixel positions received from the pixel 
position generator and compensates the received captured image brightness values by the inverse 
of the response functions of light-sensing elements having sensitivity levels corresponding to the 
received pixel positions to provide respective compensated captured image brightness values at 
20 the pixel positions received from the pixel position generator. The off-grid estimator also 
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includes a compensated on-grid brightness value memory for storing respective compensated 
captured image brightness values at respective pixel positions, an off-grid position generator for 
providing the multiplicity of off-grid positions and a pseudoinverse estimator. The 
pseudoinverse estimator receives off-grid positions from the off-grid position generator and 
derives from the compensated captured image brightness values in the compensated on-grid 
brightness value memory the respective estimated off-grid brightness values at the received off- 
grid positions. 

In a further exemplary embodiment of the system of the present invention where the 
system includes an off-grid brightness value memory for storing the respective estimated off-grid 
brightness values derived by the off-grid estimator at the multiplicity of off-grid positions and an 
on-grid resample for deriving from the estimated off-grid brightness values in the off-grid 
brightness value memory respective resampled brightness values at the multiplicity of pixel 
positions of the pixel position array , the on-grid resampler comprises a pixel position generator 
for providing the multiplicity of pixel positions, an interpolator for receiving pixel positions from 
the pixel position generator and deriving a respective resampled brightness value for each one of 
the received pixel positions by computing the product of the estimated off-grid brightness values 
at a predetermined number of off-grid positions that are nearest neighbors to the received pixel 
positions and an interpolation kernel having the same dimensions as the predetermined number 
of nearest neighbor off-grid positions. 
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In a still further exemplary embodiment of the system of the present invention, there is 
provided a system for capturing a relatively high dynamic range image of a scene using a 
relatively low dynamic range image sensor adapted to be exposed to incident light from the 
scene. The system comprising an image sensor having a multiplicity of light-sensing elements in 
5 a linear or two-dimensional array, each light-sensing element having a respective one of a 

plurality of sensitivity levels in accordance with a predetermined sensitivity pattern for the light- 
sensing element array, and being responsive to incident light from the scene for producing a 
rj captured image brightness value at a corresponding one of a multiplicity of pixel positions, 
yi where the pixel position array is rectangular, the captured image brightness values may be 
•"f 0 produced by corresponding light-sensing elements of a two-dimensional image sensor, or may be 
^'l^ a concatenation of linear brightness value arrays of line images of a scanned scene successively 
rj captured by image sensor. The system further comprises a captured image memory for storing 
lU the captured image brightness values produced by the light-sensing elements at respective pixel 

positions of a pixel position array, which may be a linear array having regularly spaced pixel 
1 5 positions in a row or a rectangular array having pixel positions located at respective intersections 
of regularly spaced orthogonal pixel rows and pixel columns. The system also comprises a pixel 
position generator for providing the pixel positions of the pixel position array and an on-grid 
interpolation filter receiving pixel positions from the pixel position generator for deriving from 
the captured image brightness values in the captured image memory interpolated on-grid 
20 brightness values at the pixel positions received from the pixel position generator. The filter 
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computes the product of respective captured image brightness values at a predetermined number 
of nearest neighbor pixel positions to each pixel position received from the pixel position 
generator and an on-grid interpolation filter kernel to derive an interpolated on-grid brightness 
value at each pixel position received from the pixel position generator, the dimensions of the on- 
5 grid interpolation filter being the same as the dimensions of the predetermined number of nearest 
neighbor pixel positions. Also included in the system is a response function compensator for 
compensating the interpolated on-grid brightness value at each one of the received pixel 
positions by the inverse of a combined response function of the light-sensing elements to derive a 
i n compensated interpolated on-grid brightness value at the received pixel position, and an output 
i^W) image memory for storing the compensated interpolated on-grid brightness values at the 
'^U respective pixel positions of the pixel position array as the pixel brightness values of a high 

dynamic range output image. The combined response function of the light-sensing elements is 
fy the sum of weighted response functions of light-sensing elements having different ones of the 
u plurality of sensitivity levels, each one of the weighted response functions being the response 
1 5 function of a light-sensing element having a respective one of the plurality of sensitivity levels 
multiplied by a predetermined weighting factor for the respective one of the plurality of 
sensitivity levels. 
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BRIEF DESCRIPTION OF DRAWINGS 

For a more complete understanding of the nature and benefits of the present invention, 
reference should be made to the following detailed description taken in conjunction with the 
accompanying drawings, in which: 

Fig. 1 is a data flow diagram representing a system for obtaining high dynamic range 
images according to the present invention; 

Fig. 2 is a data flow diagram representing the image processing part of the data flow 
diagram of Fig. 1 according to an exemplary embodiment of the present invention; 

Fig. 3 is a block diagram depicting the hardware components of an exemplary system in 
accordance with the present invention; 

Fig. 4 depicts an illustrative mask having a spatial repetition of groups of four cells where 
each cell of each group has a different transparency in a common predetermined positional order; 

Fig. 5 is a group of four cells having different transparencies in a predetermined 
positional order of the mask of Fig. 4; 

Fig. 6 illustrates the capture of a scene having a bright object and a dark object using a 
masked image sensor in accordance with the present invention; 

Fig. 7 is an illustrational representation of a captured image of a scene by a masked image 
sensor in accordance with the present invention; 

Fig. 8 is a schematic representation of the cross section of a light-sensing element and 
associated structure of a typical charge couple device image sensor; 
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Fig. 9 is a schematic representation of the cross section of two adjacent Ught-sensing 
elements and associated structure of a charge coupled device image sensor illustrating one 
technique for providing the light-sensing elements v^ith different sensitivity levels; 

Fig. 10 is a schematic representation of the cross section of two adjacent light-sensing 
5 elements and associated structure of a charge coupled device image sensor illustrating another 
technique for providing the light-sensing elements with different sensitivity levels; 

Fig. 1 1 is a schematic representation of the cross section of two adjacent light-sensing 
elements and associated structure of a charge coupled device image sensor illustrating yet 
In another technique for providing the light-sensing elements with different sensitivity levels; 
^'JO Fig. 12 is a schematic representation of a simplified charge coupled device image sensor 

1':^ having an interline structure showing terminals for applying reset trigger pulses and transfer 

trigger pulses for respective light-sensing elements to provide the light-sensing elements with 
llJ different sensitivity levels; 

Fig. 1 3 is a schematic representation of the cross section of a light-sensing element, its 
15 associated structure and a stage of a corresponding vertical shift register of a typical charge 
coupled device image sensor having an interlined structure illustrating the sinking of stored 
photogenerated charge upon the application of a reset trigger signal and the transfer of stored 
photogenerated charge to a respective shift register stage upon the application of a transfer 
trigger signal; 
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Fig. 14 show timing diagrams depicting waveforms of an exposure pulse, reset trigger 
pulses, transfer trigger pulses for four photodiodes and the magnitude of charge stored in each of 
the four photodiodes for the simplified charge coupled device image sensor depicted in Fig. 13 to 
provide different sensitivity levels for the four light-sensing elements of the image sensor; 
5 Fig. 15 is a diagram illustratively depicting pixel positions located at respective 

intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel 
columns orthogonal to the pixel rows, and off-grid positions located at respective intersections of 
p=l a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate 
yi columns; 

Fig. 16 is a data flow diagram of the off-grid estimator of the diagram of Fig. 2 according 
% to another exemplary embodiment of the present invention; 

r:i Fig. 17 is a flow diagram describing a computation process that may be carried out by the 

rU embodiment of the off^grid estimator of Fig. 1 6; 

:::f Fig. 1 8 is a flow diagram describing the computation process for evaluating the 

1 5 ResamplCj (Ic,S,x,y) function used in the computation process described by the flow diagram of 
Fig. 17; 

Fig. 19 is a flow diagram of the computation process for evaluating the Map (LUT,s) 
function used in the computation process described by the flow diagram of Fig. 17; 

Fig. 20 graphically illustrates examples of radiometric response functions of four light- 
20 sensing elements having four different sensitivity levels; 



NY02-241764 1 



- 19- 



FILE NO. A32708-070050.1 125 

Fig. 21 graphically illustrates an exemplary combined radiometric response function of 
the light-sensing elements derived by computing the weighted sum of the exemplary radiometric 
response functions of light-sensing elements having four different sensitivity levels depicted in 
Fig. 20; 

5 Fig. 22 graphically illustrates lookup table data derived from the inverse of the combined 

radiometric response function of Fig. 21; 

Fig. 23 graphically illustrates lookup table data for an inverse of a combined radiometric 
response function in which gamma characteristics are included; 
tn Fig. 24 is a data flow diagram of the off-grid estimator of the diagram of Fig. 2 according 

rf 0 to yet another exemplary embodiment of the present invention; 

'i' Fig. 25 is a flow diagram describing a computation process that may be carried out by the 

f 3 embodiment of the off-grid estimator represented by the data flow diagram of Fig. 24; 
m Fig. 26 is a flow diagram describing an alternative computation process that may be 

^-f carried out by the embodiment of the off-grid estimator represented by the data flow diagram of 
15 Fig. 24; 

Fig. 27 shows an example of a 4 x 4 cubic interpolation filter kernel; 
Fig. 28 is a data flow diagram of the on-grid resampler of the data flow diagram of Fig. 2 
according to still another exemplary embodiment of the present invention; 

Fig. 29 is a diagram of pixel positions and off-grid positions for explaining on-grid 
20 resampling using a 4 x 4 cubic interpolation filter; 
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Fig. 30 is a flow diagram describing a computation process that may be carried out by the 
embodiment of the on-grid resampler represented by the data flow diagram of Fig. 28; 
Fig. 31 shows an example of a bi-cubic B-spline interpolation filter kernel; 
Fig. 32 shows an example of a bi-linear interpolation filter kernel; 
Fig. 33 shows an example of a 6 x 6, 2-dimensional Gaussian interpolation filter kernel; 
Fig. 34 shows an example of a simple shift filter kernel; 

Fig. 35 depicts an illustrative mask having cells with two different transparencies for 
providing an image sensor with a spatially varying sensitivity pattern with two sensitivity levels; 

Fig. 36 is a diagram illustratively depicting pixel positions located at respective 
intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel 
columns orthogonal to the pixel rows, and off-grid positions located at respective intersections of 
the pixel rows and a plurality of regularly spaced intermediate columns and at respective 
intersections of the pixel columns and a plurality of regularly spaced intermediate rows; 

Fig. 37 is a data flow diagram of the off-grid estimator of the data flow diagram of Fig. 2 
for an image sensor having two sensitivity levels in accordance with a further exemplary 
embodiment of the present invention; 

Fig. 38 is a flow diagram describing a computation process that may be carried out by the 
embodiment of the off-grid estimator for two sensitivity levels represented by the flow diagram 
of Fig. 37; 
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Fig. 39 is a flow diagram describing a computation process that may be carried out by the 
exemplary embodiment represented by the data flow diagram of Fig. 24 to provide an exemplary 
embodiment of a two sensitivity level off-grid estimator with a cubic pseudoinverse estimator 
according to the present invention; 
5 Fig. 40 is a flow diagram describing a computation process for evaluating the function 

Pseudoinverse (Ij„,x',y') used in the computation process described in the flow diagram of Fig. 
39; 

n Fig. 41 is a diagram showing a shaded region of pixel positions and off-grid positions for 

\M explaining on-grid resampling using a 45 ^ rotated cubic interpolation filter kemel for an 

0 exemplary embodiment of the on-grid resampler for two sensitivity levels in accordance with the 
I'i^ present invention; 

□ Fig. 42 shows an example of a 45 rotated cubic interpolation filter kemel that may be 

ry used in the computation process described by the flow diagram of Fig. 40 for an exemplary 

embodiment of the on-grid resampler for two sensitivity levels; 
1 5 Fig. 43 is a flow diagram describing the computation process that may be carried out by 

the exemplary embodiment of the on-grid resampler represented by the data flow diagram of Fig. 

28 to obtain an exemplary embodiment of the on-grid resampler for two sensitivity levels in 

accordance with the present invention; 

Fig. 44 is a data flow diagram of still further exemplary embodiment of the image 
20 processing part of the data flow diagram of Fig. 1 ; 



NY02:241764.1 



-22- 



FILE NO. A32708-070050,l 125 

Fig. 45 is a detailed data flow diagram of yet further exemplary embodiment of the image 
processing part of the data flow diagram of Fig. 1 for four sensitivity levels; 

Fig. 46 is a flow diagram describing the computation process that may be carried out by 
the exemplary embodiment of the image processing part represented by the data flow diagram of 
Fig. 45; 

Fig. 47 is a data flow diagram of still another exemplary embodiment of the image 
processing part of the data flow diagram of Fig. 1 ; 

Fig. 48 is a flow diagram describing the computation process that may be carried out by 
the exemplary embodiment of the image processing part represented by the flow diagram of Fig. 
47; 

Fig. 49 shows an example of a 5 x 5 filter kernel that may be used in the computation 
process described by the flow diagram of Fig. 48; 

Fig. 50 illustrates the capture of successive line images of a scene by an imaging system 
having a linear image sensor, the imaging system scanning the scene by moving at constant 
speed with respect to the scene, the linear image sensor taking sequential exposures of the scene 
at regular time intervals; 

Fig. 51 illustrates the capture of successive line images of objects on a conveyor belt 
moving at constant speed past a stationary imaging system having a linear image sensor taking 
sequential exposures of the conveyor belt and objects at regular time intervals; 
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Fig. 52 depicts an illustrative exposure mask having cells with two different 
transparencies for providing a linear image sensor with a spatially varying sensitivity pattern 
with two sensitivity levels; 

Fig. 53 depicts illustratively a concatenation of the exposure mask of Fig. 52 representing 
a two-dimensional spatially varying sensitivity pattern of a two-dimensional image formed by 
concatenating successive line images captured by a linear image sensor having a one- 
dimensional spatially varying sensitivity pattern defined by the exposure mask of Fig. 52; 

Fig. 54 is a diagram illustratively depicting a linear array of pixel (on-grid) positions and 
a linear array of off-grid positions; 

Fig. 55 is a flow diagram describing a computation process that may be carried out by the 
embodiment of the off-grid estimator represented by the flow diagram of Fig. 24 for a line image 
captured by a linear image sensor having light-sensing elements of two sensitivity levels; 

Fig. 56 shows an example of a 4 x one-dimensional Gaussian interpolation kernel that 
may be used in the computation process described by the flow diagram of Fig. 55; 

Fig. 57 is a flow diagram describing a computation process that may be carried out by the 
embodiment of the on-grid resampler represented by the data flow diagram of Fig. 28 to derive 
on-grid pixel brightness values of a one-dimensional output image; 

Fig. 58 shows an example of a 4 x 1 one-dimensional cubic interpolation kernel that may 
be used in the computation process described by the flow diagram of Fig. 57; 
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Fig. 59 is a data flow diagram representing still another exemplary embodiment of the 
image processing part of the data flow diagram of Fig. 1 ; 

Fig. 60 is a flow diagram describing a computation process that may be carried out by the 
embodiment of the image processing part represented by the data flow diagram of Fig. 59 to 
derive on-grid pixel brightness values of a linear output image; and 

Fig. 61 shows an example of a 5 x 1 on-grid interpolation filter kernel that may be used in 
the computation process described by the flow diagram of Fig. 60. 

Throughout the figures of the drawings the same reference numerals and characters are 
used to designate common features, components, parts, processes and steps. 

DETAILED DESCRIPTION 

Referring to Fig. 1, there is shown a data flow diagram 100 of a system for capturing 
relatively high dynamic range images using a relatively low dynamic range image sensor in 
accordance with the present invention. The system has an image sensing part 1 that includes a 
relatively low dynamic range image sensor (not shown in Fig. 1) having a multiplicity of light- 
sensing elements in an array (not shown). The term "light" as used in the specification and 
claims shall mean electromagnetic radiation whose wavelength may be anywhere in the spectrum 
from the far infrared to x-rays. The array of light-sensing elements of the image sensor may have 
any arrangement, including vdthout limitation rectangular, linear, circular and hexagonal grids, 
and the light-sensing elements in the array need not be uniformly disposed. The array of light- 
sensing elements has a predetermined spatially varying sensitivity pattern to incident light, and 
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each light-sensing element of the image sensor produces in response to incident light from a 
scene a captured image brightness value at a corresponding one of a multiplicity of pixel 
positions of a pixel position array defined by respective intersections of a plurality of regularly 
spaced pixel rows and a plurality of regularly spaced pixel colunms orthogonal to the pixel rows 
5 (not shown in Fig. 1). The captured image brightness values at respective pixel positions 

produced by the light-sensing elements are stored in a captured image memory (not shown in 
Fig. 1) in the image processing part 2 of the system. The image processing part 2 serves to 
q: remove the variation in image brightness caused by the spatially varying sensitivity pattern of the 
IM image sensor in the image sensing part 1 and provides resampled brightness values at respective 
"ft pixel positions, representing a relatively high dynamic range output image, to the output image 
^1.^ memory 3. 

1:3 Turning to Fig. 2, there is shown a dataflow diagram 200 of the image processing part 2 

rU of the data flow diagram of Fig. 1. The captured image brightness values at respective pixel 
positions of the pixel position array, as provided by the image sensor, are stored in a captured 

15 image memory 20. The captured image brightness values in the captured image memory 20 are 
used by an off-grid estimator 22 to derive estimated off-grid brightness values at off-grid 
positions located at respective interstices of the pixel position array. The estimated off-grid 
brightness values at respective off-grid positions are then resampled by an on-grid resampler 23 
to derive resampled brightness values at respective pixel positions. The resampled brightness 

20 values are stored in an output image memory 21 as the pixel brightness value representing the 
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relatively high dynamic range image. Because the brightness value produced by each one of the 
light-sensing elements of the image sensor is not combined with the brightness value produced 
by any other light-sensing element, there is no reduction in the resolution of the image captured 
by the image sensor. Also, because the brightness values at respective off-grid positions are 
estimated from the captured image brightness values at respective pixel positions and the 
resampled brightness values at respective pixel positions are derived by resampling of the 
estimated off-grid brightness values at the off-grid positions, the captured image brightness value 
produced by any one of the light-sensing elements can contribute to any one of the resampled 
brightness values. Consequently, the resampled output image in the output image memory 21 
has almost the same resolution as the captured image in the captured image memory 20. 

Referring to Fig. 3, there is shown a block diagram of the hardware components of a 
system 300 according to an exemplary embodiment of the present invention. The system 300 
has an image sensing part 1, an image processing part 2 and video output part 4. A scene is 
captured by the image sensing part 1 with an image sensor having a predetermined spatially 
varying sensitivity pattern. The image sensing part 1 provides an image signal representing the 
captured image to the image processing part 2. The image processing part removes the variations 
in brightness of the image caused by the spatially varying sensitivity pattern of the image sensor 
and provides an output image signal to the video output part 4. The video output part 4 is not a 
part of the present invention but is included here for the purpose of showing a complete video 
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imaging system that provides an output video signal representing relatively high dynamic range 
images. 

The image sensing part 1 includes a lens system 6, an aperture 7, a charge coupled device 
(CCD) image sensor 9 and a preamplifier 10. Incident light 5 from a scene passes through the 
lens system 6 and the aperture 7 before reaching the CCD image sensor 9. The CCD image 
sensor 9 has a predetermined spatially varying sensitivity pattern according to which the 
sensitivity level of each of the light-sensing elements (not shown) of the image sensor 9 is 
determined. Different exemplary ways of providing a spatially varying sensitivity pattern for the 
CCD image sensor 9 are described hereinbelow. The preamplifier 10 consists of a group of 
several analog circuits which are typically used in video cameras, such as an automatic gain 
control circuit for adjusting the amplitude of the CCD image sensor output and a correlated 
double sampling circuit for reducing noise. 

The image sensing part 2 has an analog-to-digital (A/D) converter 1 1 , a frame memory 
12, a digital-to-analog (D/A) converter 13, a read-only memory (ROM) 14, a random-access 
memory (RAM) 15 and a processor unit 16. The A/D converter 1 1 digitizes the image signal 
from the image sensing part 1, and the digitized image data (captured image brightness values at 
respective pixel positions) are stored in the frame memory 12. Computations in the image 
processing part 2 are performed by the processing unit 1 6, which has access to the data in the 
frame memory 12, the ROM 14 and the RAM 15. Predefined data and programs to be used by 
the processing unit 16 to perform its computations are stored in the ROM 14. A portion of the 
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RAM 15 is used for temporary storage of data diiring computations by the processing unit 16. 
Computed resampled brightness values at respective pixel positions, which are stored in the 
frame memory 12, are converted to an analog image signal by the D/A converter 13 and provided 
to the video output part 4. 

The video output part 4 comprises a video encoder 17 and an output unit 18. The analog 
image signal from the image processing part 2 is encoded to a video signal by the video encoder 
17, and the output unit 18 outputs the video output signal 19. 
Image Sensing Part 

The spatially varying sensitivity pattern of the CCD image sensor 9 may be implemented 
by using a fixed pattern mask or by on-chip techniques. Turning now to Fig. 4, there is shown an 
example of a fixed pattern mask 101 having a predetermined spatially varying exposure pattern. 
In this example, the fixed pattern mask 101 is a plate having a two-dimensional array of cells of 
different transparencies (attenuations). The exposure pattern of the mask 101 is defined by a 
repetitive pattern of identical groups of four nearest neighbor cells each having a different 
transparency. An example of such a group of four nearest neighbor cells is shown in Fig. 5. The 
group has a most transparent cell 102, a transparent cell 103, a less transparent cell 104 and a 
least transparent cell 105 arranged in a predetermined positional order common to all groups of 
cells of the mask 101. Each cell of the mask 101 is used to control the sensitivity level of a 
respective light-sensing element of the image sensor. In the case of a CCD image sensor, the 
mask 101 may be fabricated on the same substrate as the CCD device by forming a transparent 
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layer over the photosensitive region of the device and forming the cells of different 
transparencies by patterning the layer using known photolithography and etching techniques to 
form a mask with a desired exposure pattern. Alternatively, the light-sensing surfaces of the 
light-sensing elements themselves may be subjected to masked etching operations to alter the 
5 photosensitivities of preselected light-sensing elements to desired levels. By using the fixed 

pattern mask 101, each local area of a captured image exposed through one of the groups of four 
neighboring cells 102, 103, 104 and 105 will have four different exposures. In this manner, 
every such local area of the captured image will have a relatively high dynamic range resulting 

m from the use of four different exposures (i.e., four light-sensing elements having different 

^TO sensitivity levels). 

■"^ Referring to Fig. 6, there is illustrated an arrangement for capturing the image of a scene 

r j using the fixed pattem mask 101. As shown in the figure, the fixed pattern mask 101 is placed 
i'il directly in front of an image sensor 108. The mask 101 is aUgned with the image sensor 108 so 

that each cell of the mask 101 is positioned directly in front of a corresponding light-sensing 
1 5 element (not shown) of the image sensor 108. Incident light from the scene passes through the 
camera optics 109 and is then focused onto the image plane of the image sensor 108 through the 
mask 101 . The brightness of the incident light which is detected by each light-sensing element 
of the image sensor 108 is determined by the transparency of the mask cell which is in front of 
that light-sensing element. In the scene which is being captured in Fig. 6, there is a bright object 
20 106 and dark object 107. 
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Tiiming to Fig. 7, there is shovm a representation 1 10 of the image of the scene in Fig. 6 
as captured by the image sensor 108 through the fixed pattern mask 101 , It is noted in Fig. 7 that 
even if pixels of brighter exposure have saturated brightness values in an area of the bright object 
106, pixels of darker exposure in the same area have non-saturated brightness values. Similarly, 
5 even if pixels of darker exposure have brightness values below the noise level of the light- 
sensing elements receiving incident light from the area of the dark object 107 in Fig. 6, pixels of 
brighter exposure receiving incident light from the same area will have brightness values above 
fj the noise level. 

m Referring to Fig. 8, there is shown in schematic representation 800 of the cross section of 

^fO a light-sensing element and associated structure of a typical CCD image sensor, which is the 
preferred type of image sensor for implementing a system according to the present invention. 
n The light-sensing element comprises a photodiode consisting of an n-type region 1 1 5 formed 
rU within a p-well region 190. The photodiode has a light-sensing sxarface 191 adapted to be 

exposed to incident light and is operatively biased (n-type region 115 being reverse biased with 
15 respect to the p-well region 190) to form a potential well in the p-well region for storing 

photogenerated charge therein. Associated with the photodiode is a transfer gate electrode 1 14, 
which upon the application of an appropriate voltage signal thereto creates a channel for the 
photogenerated charge stored in the photodiode to pass to a vertical shift register (not shown in 
Fig. 8), as will be further explained hereinbelow. An opaque light shield layer 1 13 is formed to 
20 prevent incident light from reaching regions other than the light-sensing surfaces of the 
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photodiodes of the CCD image sensor. An aperture 192 is formed in the light shield layer 1 13 to 
expose the Ught-sensing surface 191 of each one of the photodiodes in the CCD image sensor. 
An on-chip micro lens 1 1 1 may be formed above the Ught-sensing surface 191 of the photodiode 
to concentrate incident Ught onto the Ught-sensing surface 191. EspeciaUy for color CCD image 
sensors, there is formed an on-chip optical filter 1 12 between the on-chip micro lens 1 1 1 and the 
light-sensing surface 191 of the photodiode. 

Turning to Fig, 9, there is shown a schematic representation 900 of the cross section of 
two adjacent light-sensing elements and associated features of a CCD image sensor, in which the 
two light-sensing elements have different sensitivity levels. The light-sensing element on the left 
of the figure has an on-chip micro lens 1 1 1 for concentrating incident light onto the light 
sensitive surface 191 of photodiode, while the light-sensing element on the right does not have 
an on-chip micro lens above the light-sensing surface 191 of its photodiode. Therefore, the light- 
sensing element on the left will receive a greater intensity of incident light at its light-sensing 
surface 191 than the light-sensing element on the right. In this manner the light-sensing element 
on the left is fabricated to have a higher sensitivity level than the light-sensing element on the 
right. By forming an on-chip micro lens 1 1 1 above preselected light-sensing elements of the 
image sensor a predetermined spatially varying sensitivity pattern having two sensitivity levels is 
attained for the CCD image sensor . 

Turning to Fig. 10, there is a shown a schematic representation 1000 of the cross section 
of two adjacent light-sensing elements and associated structure of a CCD image sensor. 
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illustrating an alternative technique for providing the light-sensing elements with different 
sensitivity levels. Respective on-chip filters 116 and 1 17 are formed between the on-chip micro 
lenses 111 and the light-sensing surfaces 191 of the right and left photodiodes, respectively. The 
filters 116 and 1 17 are formed to have different transparencies to incident light, thus causing the 
5 light-sensing elements on the right and left to have different sensitivity levels. By forming the 
filters above the photosensitive surfaces 191 of respective photodiodes of the CCD image sensor 
to have predetermined transparencies, a predetermined spatially varying sensitivity pattern is 
r=j attained for the CCD image sensor. 

Ul Referring to Fig. 1 1 , there is shown a schematic representation 1 100 of the cross section 

i |0 of two adjacent light-sensing elements and associated structure of a CCD image sensor. The 
''11 light shields layer 1 13 has respective apertures 118 and 1 19 above the photosensitive surfaces 

191 of the left and right light-sensing elements. The apertures 118 and 1 19 in the light shield 
r U layer 1 1 3 are formed to have different sizes so as to allow different amounts of incident light to 

reach the light-sensing surfaces 191 of the left and right photodiodes. By forming the apertures 
15 above the light-sensing surfaces 191 of respective light-sensing elements of the image sensor to 

have predetermined sizes, a predetermined spatially varying sensitivity pattern is attained for the 

image sensor. 

Referring to Fig. 12, there is shown a schematic layout diagram 1200 representing a 
simplified CCD image sensor having four photodiodes 121, 122, 123 and 124, and an interline 
20 structure in which each column of photodiodes 121 and 123, and 124 and 122 is located adjacent 
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a corresponding vertical shift register 125 and 126, respectively. As explained above in 
connection with Fig. 8, each photodiode has an associated transfer gate electrode 1 14. When an 
appropriate voltage transfer trigger signal is applied to the transfer gate electrode 1 14 associated 
with a photodiode, photogenerated charge stored in the photodiode is allowed to pass to a 
5 respective stage of the corresponding vertical shift register and is accumulated therein for the 
duration of an exposure period. The diagram of Fig. 12 shows four transfer trigger inputs 129, 
130, 131 and 132 for applying transfer trigger signals to the transfer gate electrodes associated 
with photodiodes 121, 122, 123 and 124, respectively, to transfer photogenerated charge stored 

Ln in the photodiodes 121, 122, 123 and 124 to respective stages of corresponding shift registers 

rK) 125 and 126. At the end of each exposure period stored photogenerated charge transferred from 
the photodiodes and accumulated in respective stages of the vertical shift registers are shifted in 
each vertical shift register 125 and 126 by a three-phase clock (not shown) to respective stages of 

m a horizontal shift register 127. Once in the horizontal shift register 127, the accumulated 

photogenerated charge fi-om each one of the photodiodes 121, 122, 123 and 124 are shifted out 

1 5 sequentially by a two phase clock (not shown) to an output terminal 1 28 where the respective 
charge packets are detected as voltage signal proportional to the magnitude of the accumulated 
photogenerated charge packet reaching the output terminal 128. 

Turning to Fig. 13, there is shown a schematic representation 1300 of the cross section of 
a photodiode formed by n-type region 1 15 and p-well region 190, a stage of the corresponding 

20 vertical shift register formed by n-type region 135 and p-well region 190, a p-type barrier region 
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134 separating the photodiode and the shift register stage, and the transfer gate electrode 1 14 
associated with the photodiode connected to a transfer trigger input 137. The shift register stage 
comprises an n-type region 135 which is appropriately biased with respect to the p-well 190 to 
form a potential well (not shown) for receiving and accumulating photogenerated charge 
transferred to the shift register stage. When a transfer trigger pulse signal is applied to terminal 
137, photogenerated charge stored in the potential well of the photodiode form by n-type region 
115 and the p-well 190 is allowed to pass through a channel beneath the barrier region 134 to the 
potential well of the shift register stage, as indicated by the arrow 138. 

In addition to the periodic application of a transfer trigger pulse to transfer trigger input 
137, the photogenerated charge stored in each photodiode is periodically discharged by the 
periodic application of a reset pulse signal to a reset input 133. The application of each reset 
pulse voltage signal appropriately biases the junction between the p-well and a heavily doped 
n-type charge sink region 120 to cause substantially all of the photogenerated charge stored in 
the potential well of each photodiode of the CCD image sensor to be transferred to the charge 
sink region 120, as indicated by the arrow 139. 

Turning now to Fig. 14, there are shown timing diagrams of an exposure pulse 193, reset 
trigger pulses 194, transfer trigger pulses 195, 198, 261 and 264 associated with the four 
photodiodes of the exemplary CCD image sensor of Fig. 12, and the magnitudes of the stored 
photogenerated charge 196, 199, 262 and 265 in the four photodiodes PD4, PD3, PD2 and PDl, 
respectively, of the exemplary CCD image sensor. The duration of the exposure period for 
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capturing an image is determined by the width of the exposure pulse 193. Each time a reset 
pulse 194 is applied to the reset trigger input 133 of the CCD image sensor, the photogenerated 
charge stored in each photodiode of the image sensor goes to zero for reasons explained 
hereinabove. Because all photodiodes in the CCD image sensor are exposed to incident Ught the 
5 magnitude of the photogenerated charge, as represented by waveforms 196, 199, 262 and 265, 
increases between reset pulses 1 94. As explained above, each time a transfer trigger pulse is 
applied to the transfer gate electrode associated with a photodiode, photogenerated charge stored 
in the photodiode is transferred to a respective stage of a corresponding vertical shift register. In 
i^fi the case of photodiode PD4, transfer trigger pulses 195 are applied to its associated transfer gate 
i40 electrode at the same frequency as the reset pulses 194 but with the phase shifted so that each 

transfer trigger pulse 195 immediately precedes the next reset pulse 194. The application of each 
rj transfer trigger pulse 195 causes an amount of charge represented by a corresponding blackened 
fU region 197 to be transferred to a respective stage of a corresponding vertical shift register, and 

the transferred charge is accumulated therein for the duration of the exposure pulse 193. 
1 5 Transfer trigger pulses 1 98, which have the same width as transfer trigger pulses 1 95, are 

applied to the transfer gate electrode associated with photodiode PD3 at half the frequency as that 
of transfer trigger pulses 195. The phase of the transfer trigger pulses 198 is such that each 
transfer trigger pulse 198 immediately precedes the next reset pulse 194. The application of each 
transfer trigger pulse 198 causes an amount of photogenerated charge stored in photodiode PD3 
20 represented by the corresponding blackened region 260 to be transferred to a respective stage of a 
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corresponding vertical shift register, and the transferred charge is accumulated therein for the 
duration of the exposure pulse 193. Because transfer trigger pulses 198 occur at half the 
frequency of the transfer trigger pulses 195, the photogenerated charge accumulated in the 
vertical shift register stage receiving photogenerated charge fi-om photodiode PD3 over the 
exposure duration will be half of that which is accumulated in the vertical shift register stage that 
receives photogenerated charge from photodiode PD4 over the same exposure duration. 
Operated in this maimer, the effective sensitivity level of photodiode PD3 will be half that of 
photodiode PD4. 

Transfer trigger pulses 261, which have the same width as the transfer trigger pulses 198 
and 195, are applied to the transfer gate electrode associated with photodiode PD2. The phase of 
the transfer trigger pulses 261 is such that each transfer trigger pulse 261 immediately precedes 
the next reset pulse 194. The application of each transfer trigger pulse 261 causes the transfer of 
an amount of photogenerated charge stored in photodiode PD2 represented by the corresponding 
blackened region 263 to a respective stage of a corresponding vertical shift register, and the 
transferred charge is accumulated therein for the duration of the exposure pulse 193. Because the 
frequency of the transfer trigger pulses 261 is half that of the transfer trigger pulses 198 and one- 
quarter of that of the transfer trigger pulses 195, the total amount of photogenerated charge 
accumulated in the shift register stage that receives stored photogenerated charge from 
photodiode PD2 over the duration of the exposure pulse 193 will be half of that which is 
accumulated in the vertical shift register stage that receives stored photogenerated charge from 
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photodiode PD3 and one-quarter of that which is accumulated in the shift register stage that 
receives stored photogenerated charge from photodiode PD4 over the same exposure duration. 
Consequently, the effective sensitivity level of photodiode PD2 is half of the effective sensitivity 
level of photodiode PD3 and one-quarter of the effective sensitivity level of photodiode PD4. 

Transfer trigger pulses 264, which have the same width as transfer trigger pulses 261, 198 
and 195, are applied to the transfer gate electrode associated with photodiode PDl at half the 
frequency of that of transfer trigger pulses 261 . The phase of the transfer trigger pulses 264 is 
such that each transfer trigger pulse 264 immediately precedes the next reset pulse 194. The 
application of each transfer trigger pulse 264 causes an amount of photogenerated charge stored 
in photodiode PDl represented by the corresponding blackened region 266 to be transferred to a 
respective stage of a corresponding vertical shift register, and the transferred charge is 
accumulated therein for the duration of the exposure pulse 193. Because the frequency of the 
transfer trigger pulses 264 is half of that of transfer trigger pulses 261, one-quarter of that of 
transfer trigger pulses 198 and one-eighth of that of transfer trigger pulses 195, the total 
photogenerated charge accumulated in the vertical shift register stage receiving stored 
photogenerated charge from photodiode PDl over the duration of the exposure pulse 193 is one- 
half of that which is accumulated in the shift register stage receiving stored photogenerated 
charge from photodiode PD2, one-quarter of that which is accumulated in the shift register stage 
receiving stored photogenerated charge from photodiode PD3 and one-eighth of that which is 
accumulated in the vertical shift register stage which receives stored photogenerated charge from 
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photodiode PD4 over the same exposure duration. Consequently, the effective sensitivity level 
of photodiode PDl is one-half of that of photodiode PD2, one-quarter of that photodiode PD3 
and one-eighth of that photodiode PD4. Accordingly, different effective sensitivity levels may 
be obtained for the photodiodes of the CCD image sensor by applying respective transfer trigger 
pulses to their associated transfer gate electrodes at different frequencies in the manner described. 

Because photogenerated charge is accumulated in small amovmts in respective vertical 
shift register stages upon the application of each transfer trigger pulse and the transfer trigger 
pulses are uniformly distributed over the exposure period, movement in the scene will be 
integrated in the same manner in the vertical shift register stages accumulating stored 
photogenerated charge from respective photodiodes. Thus, inconsistency of motion blur in the 
output images is suppressed by using the foregoing technique for obtaining a predetermined 
spatially varying sensitivity pattern for the CCD image sensor. 
Image Processing Part 

Returning to the dataflow diagram of Fig. 2, the image processing part 200 according to 
an exemplary embodiment of the present invention comprises a captured image memory 20, an 
off-grid estimator 22, an on-grid resampler 23 and an output image memory 21. The captured 
image memory 20, which is physically implemented, for example, as part of the frame memory 
12 in the block diagram of Fig. 3, stores in digitized form the image captured by the image 
sensing part 1 of Fig, 1 . The off-grid estimator 22 reads the captured image brightness values in 
the captured image memory 20 and computes brightness values at respective off-grid positions. 
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the off-grid positions being positions which are not on the image (pixel position) array. The on- 
grid resampler 23 receives the brightness values at respective off-grid positions and computes 
resampled brightness values at respective pixel positions by resampling the off-grid brightness 
values at respective off-grid positions. The computed resampled brightness values at respective 
5 pixel positions are stored in the output image memory 21 as pixel brightness values of a 
relatively high dynamic range image of the scene. 

Referring to Fig. 15, there is shown an exemplary arrangement 1500 of pixel positions 
represented by rectangles 140 located in a pixel position array defined by respective intersections 
Lfi of a plurality of regularly spaced pixel rows 267 and a plurality of regularly spaced pixel 
ilp columns 268 orthogonal to the pixel rows 267. The off-grid positions are generally located at 

respective interstices of the pixel position array form an off-grid position array. In the 
rj arrangement of Fig. 15, the off-grid positions, which are represented by black dots 141, form an 
ilJ off-grid position array defined by respective intersections of a plurality of regularly spaced 

intermediate rows 269 and a plurality of regularly spaced intermediate columns 270 orthogonal 
1 5 to the intermediate rows 269. Each one of the intermediate rows 269 is parallel to the pixel rows 
267 and extends medially between a respective adjacent pair of the pixel rows 267. Each one of 
the intermediate colunms 270 is parallel to the pixel columns 268 and extends medially between 
a respective adjacent pair of the pixel columns 267. For convenient reference of the pixel 
positions and off-grid positions, the bottom left comer of the captured image is defined as the 
20 origin of a coordinate system (0,0) with the pixel position nearest the origin having coordinates 
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(0.5, 0.5) and the off-grid position nearest the origin having coordinates (IJ). The bottom right 
comer of the captured image is defined as having coordinates (xSize, 0), where xSize is the 
dimension of the captured image in the direction (x direction) of the pixel rows 267, and the 
upper left comer of the image is defined as having coordinates (0,ySize), where ySize is the 
dimension of the captured image in the direction (y direction) of the pixel columns 268. 

Turning to Fig. 16, there is shown a dataflow diagram 1600 of the off-grid estimator (22 
in the diagram of Fig. 2) according to an exemplary embodiment of the present invention. For 
this embodiment the light-sensing elements of the image sensor have four different sensitivity 
levels. The off-grid estimator 1600 has four resamplers 28, 29, 30 and 31, each corresponding to 
a respective one of the four sensitivity levels, an off-grid position generator 32 for providing the 
off-grid positions of the off-grid position array as defined by the diagram of Fig. 15, and a 
sensitivity pattern data memory 34 for storing data indicative of the sensitivity level 
corresponding to each one of the pixel positions. In addition, the off-grid estimator of 1600 
includes weighting factor memories 24, 25, 26 and 27 for storing weighting factors W^, W2, W3 
and W4, respectively, each weighting factor corresponding to a respective one of the sensitivity 
levels of the light-sensing elements, and multipliers 271, 272, 273 and 274, each of which 
receive the output of a respective one of resamplers 28, 29, 30 or 3 1 and weighting factors Wi, 
W2, W3, or W4 corresponding to the same sensitivity level as the respective one of the 
resamplers, and providing the product of the received resampler output and weighting factor. 
The off-grid estimator 1600 also includes an adder 275 which receives the outputs of multipliers 
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271, 272, 273 and 274 and provides a sum of the multiplier outputs. Furthermore, the off-grid 
estimator 1600 has a mapper 33, which receives the output of the adder 275, a lookup table 
memory 35 associated with the mapper 33, The lookup table memory 35, which stores data 
representing the inverse of a sensitivity weighted combined response function of light-sensing 
elements having different ones of the plurality of sensitivity levels, is used by the mapper 33 to 
compensate the output of the adder 275 for each off-grid position received from the off-grid 
position generator 32 by the inverse of the combined response function to derive an estimated 
off-grid brightness value at the received off-grid position. The off-grid estimator 1600 also has 
an off-grid brightness value memory 36 for receiving the output of the mapper 33. 

Referring again to Fig. 3, the sensitivity pattern memory 34 in Fig. 16 is physically 
implemented, for example, in the ROM 14 and stores the predetermined spatially varying 
sensitivity pattern of the CCD image sensor 9 in the form of a respective sensitivity level for 
each one of the pixel positions. The lookup table memory 35 in Fig. 16 is also physically 
implemented, for example, in the ROM 14 and stores a lookup table for compensating the output 
of the adder 275 at each one of the off-grid positions by the inverse of a sensitivity weighted 
combined response function of the light-sensing elements of the image sensor. The off-grid 
brightness value memory 36 in Fig. 16 is physically implemented, for example, in the frame 
memory 12 and stores estimated off-grid brightness values at respective off-grid positions as 
provided by the mapper 33 in Fig. 16. 
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Tviming back to Fig. 16, the captured image memory 20 stores captured image brightness 
values at respective pixel positions defined in accordance with the diagram of Fig. 15. The 
brightness value at each pixel position is produced, for example, by a corresponding light- 
sensing element of the CCD image sensor 9 of the block diagram of Fig. 3. Because each light- 
5 sensing element has one of the four sensitivity levels and produces a brightness value at a 

corresponding pixel position, each pixel position can be considered as having a corresponding 
sensitivity level. Each one of the resamplers 28, 29, 30 and 31 receives an off-grid position from 
the off-grid position generator 32 and sensitivity pattern data from the sensitivity pattern memory 
i S 34, and derives a sensitivity level off-grid brightness value for its corresponding sensitivity level 
\rM) at the received off-grid position by resampling the captured image brightness values in the 

captured image memory 20 at pixel positions corresponding to that sensitivity level. The details 
:L=. of the resampling computation are described hereinbelow. After each one of the resamplers 28, 
fy 29, 30 and 31 derives a respective sensitivity level off-grid brightness value for its corresponding 
Q sensitivity level at an off-grid position received from the off-grid position generator 32, each one 
15 of the derived sensitivity level off-grid brightness values is multiplied by a respective one of 

weighting factors Wj, W2, W3, or W4 corresponding to the same sensitivity level as the deriving 
resampler in one of the multipliers 271, 272, 273 and 274. The sensitivity level off-grid 
brightness values derived by the four resamplers 28, 29, 30 and 3 1 at the off-grid position 
received from off-grid position generator 32 after scaling by respective weighting factors Wj, 
20 W2, W3, or W4 are then accumulated by the adder 275 to derive a weighted sum sensitivity level 
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off-grid brightness value at the received off-grid position, which is provided to the mapper 33. 
The mapper 33 rounds off the weighted sum sensitivity level off-grid brightness value from the 
adder 275 to obtain an integer index for the lookup table memory 35. A combined response 
function compensated brightness value corresponding to the index is retrieved by the mapper 33 
from the lookup table memory 35 and provided to the off-grid brightness value memory 36 as the 
estimated off-grid brightness value at the off-grid position received from the off-grid generator 
32. Since the off-grid position generator 32 sequentially generates off-grid positions from (1,1) 
to (xSize- 1, ySize- 1), an off-grid brightness value pattern with dimensions xSize- 1 and 
ySize- 1 is generated and stored in the off-grid brightness value memory 36. 

Referring to Fig. 17, there is shown a flow diagram of the computation process that may 
be carried out by the off-grid estimator 1600 of Fig. 16. First, loop 201 is repeated for each off- 
grid position (x, y) provided by the off-grid position generator 32 of Fig. 16 from (1, 1) to 
(xSize-1, ySize-1), where x and y are the coordinates of the off-grid position and xSize and ySize 
are the dimensions of the captured image in the x and y directions, respectively. In the loop 201, 
steps 202, loop 203 and step 206 are carried out. In step 202 a value s is initialized to 0. In loop 
203 steps 204 and 205 are repeated for each i-th sensitivity level. In the present embodiment, the 
number of sensitivity levels is 4, and, therefore, loop 203 is repeated four times. In step 204, a 
sensitivity level off-grid brightness value Pi for the i-th sensitivity level at off-grid position (x,y) 
is computed from the captured image brightness values 1^ at respective pixel positions stored in 
the captured image memory 20 of Fig. 16 by using a resampling function Resamplei(Ic, S,x,y), 
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where S represents the sensitivity pattern data from the sensitivity pattern memory 34 in the 
diagram of Fig. 16. The details of the resampling function are described hereinbelow. In step 
205, the computed sensitivity level off-grid brightness value for the i-th sensitivity level p^ is 
multiplied by a predetermined v^eighting factor W| corresponding to the i-th sensitivity level, and 
the product is added to the value s. After loop 203 is repeated for each sensitivity level, step 206 
is performed. In step 206 estimated off-grid brightness value B at off-grid position (x,y) is 
computed by the function Map (LUT,s) using the mapper 33 and the lookup table memory 35 in 
the diagram of Fig. 16. The details of this fimction are described hereinbelov^. After step 206 is 
finished, the current repetition of loop 201 is completed. After loop 201 is repeated for each off- 
grid position received from the off-grid position generator 32 in Fig. 16, the computation process 
represented by the flow chart 1700 terminates. 

Turning to Fig. 18, there is shown a flow diagram 1800 describing the computation of the 
fimction Resamplei(Ic, S,x,y) used in the computation process described by the flow diagram 
1700 of Fig. 17. The computation described by the flow diagram 1800 is performed by the i-th 
resampler in Fig. 16 corresponding to the i-th sensitivity level. In step 207 an image I^ having 
brightness values only at pixel positions corresponding to the i-th sensitivity level is generated by 
masking the captured image le in accordance with the sensitivity pattern data S. In step 208 an 
off-grid brightness value p at a pixel position of the masked image 1^ which is nearest to the off- 
grid position (x,y) is determined. Finally, in step 209, the brightness value p is provided as the 
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sensitivity level off-grid brightness value for the i-th sensitivity level at off-grid position (x, y), 
and the computation process represented by flow diagram 1800 terminates. 

Turning now to Fig. 19, there is shown a flow diagram 1900 for the computation of the 
function Map(LUT,s) used in the computation process described by flow diagram 1700 of Fig, 
17. The computation of flow diagram 1900 is performed by the mapper 33 and associated 
lookup table memory 35 in the diagram of Fig. 16. The computation consists of a sequence of 
steps 210, 21 1 and 212. In step 210 an integer value index t is computed by rounding off s, as 
computed in loop 203 of the flow diagram 1700 of Fig. 17. In step 211a value p is retrieved 
from the lookup table memory 35 in Fig. 16 using the index t. In step 212, the value p 
corresponding to index t in the lookup table memory 35 is provided as the output, and the 
process represented by the flow diagram 1900 terminates. The lookup table data which is used to 
map the index t to the value p is prepared in advance and stored in the lookup table memory 35 in 
Fig. 16. As indicated above, the lookup table memory 35 is implemented, for example, in the 
ROM 14 in Fig. 3. 

Referring to Fig. 20, there is shown a graph 2000 of exemplary normalized radiometric 
response functions of light-sensing elements with different ones of the four sensitivity levels. 
The quantity E represents the radiation intensity incident on the light-sensing element normalized 
by the maximum radiation intensity that can be sensed by the light-sensing element in order to 
create the graph 2000 of Fig. 20. The quantity p represents the brightness value at the pixel 
position corresponding to the light-sensing element. Since the present embodiment has four 
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sensitivity levels, there are four response functions pi (E) 142, P2(E) 143, PsCE) 144 and P4(E) 
145. Since all four response functions are monotonic, a linear combination of these functions 
s(E) with positive v^eighting factors is also monotonic. The sensitivity weighted combined 
response function of the light-sensing elements having the four sensitivity levels s(E) may be 
expressed as 

The combining represented by equation (1) brings about a high dynamic range for sensing the 
radiation intensity E because the response functions at four sensitivity levels are combined. For 
example, pi(E) has sensitivity to a bright range of E and p4(E) has sensitivity to a dark range of 
E, while P2(E) and PsCE) cover the mid range of E. The combining of equation (1) also results in 
an increase in the number of grey scale levels. If each of the response functions Pi(E), P2(E), 
P3(E) and p4(E) has a grey scale of 256 levels, s(E) has a grey scale equivalent to 1021 levels. 

Turning to Fig. 21, there is shown a graph 2100 of an exemplary combined response 
function s(E). To store lookup table data for the combined response function s(E), the inverse of 
s(E), which is shown as curve 147 in the graph 2200 of Fig. 22, can be used because the inverse 
of a monotonic function is also monotonic. If the inverse of s(E) is generated as a continuous 
function, a lookup table having any number of levels can be digitized and stored. For example, a 
lookup table derived from the graph of Fig. 22 has 65,536 levels. Moreover, data representing 
any kind of response function can be stored in a lookup table. For example, Fig. 23 shows a 
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graph 2300 of the inverse of a response function in which a gamma characteristic is included. 
Inclusion of the gamma characteristic may be necessary to provide a standardized video signal 
output. 

It v^ill be apparent to those skilled in the art that the nearest neighbor sampling function 
carried out by the resamplers 28, 29, 30 and 31 of Fig. 16 may be substituted with other well 
known sampling functions, such as bi-linear and bi-cubic sampling with interpolation. The 
weighting factors W^, W2, W3 and W4 can each be any positive value. In most instances, each of 
the weighting factors can be set to 1/N, where N is the number of sensitivity levels. In the 
embodiment of the off-grid estimator shown in Fig. 16, if the weighting factors Wi, W2, W3 and 
W4 are all equal, the weighting factor memories 24, 25, 26 and 27, and the multipliers 271, 272, 
273 and 274 may be eliminated, in which case the outputs of the resamplers 28, 29, 30 and 31 are 
provided directly to the adder 275. It is noted that the weighting factors may be selected to 
obtain a desired shape for the combined response function s(E) defined by equation (1) for 
purposes of creating the lookup table data stored in the lookup table memory 35 in the diagram 
of Fig. 4. The lookup table data is derived by digitizing the inverse of the function s(E) as shown 
in the graph of Fig. 22. To minimize the digitizing error, the inverse of the function s(E) is 
digitized as uniformly as possible. Where the shape of the inverse of the function s(E) is not well 
suited for uniform digitization, the shape may be changed by changing the weighting factors. 

Turning now to Fig. 24, there is shown a data flow diagram 2400 of the off-grid estimator 
22 of the diagram of Fig. 2 according to another exemplary embodiment of the present invention. 
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The off-grid estimator of Fig. 24 includes a pixel position generator 43 for sequentially 
generating pixel positions from (0.5,0.5) to (xSize, ySize), a sensitivity pattern memory 40 for 
storing a sensitivity level corresponding to each pixel position, a mapper 42 and associated 
lookup table memory 39 for storing separate lookup table data representing the inverse of the 
5 response functions of light-sensing elements having different ones of the sensitivity levels, a 

mapped pixel position brightness value memory 41 for storing the brightness value at respective 
pixel positions after compensation of each pixel position brightness value by the inverse of the 
response fimction of a light-sensing element having a sensitivity level corresponding to the pixel 
i Jl position, an off-grid position generator 45 for sequentially generating off-grid positions from 
hW) (1,1) to (xSize- 1, ySize- 1), a pseudoinverse estimator for computing from the compensated 
I j.^ pixel position brightness values stored in the mapped pixel position brightness value memory 41 
.-^ an estimated off-grid brightness value at each off-grid position received from the off-grid 
ry position generator 45, and an off-grid brightness value memory 36 for storing estimated off^grid 
□ brightness values at respective off-grid positions computed by the pseudoinverse estimator 44. 
1 5 With reference to the block diagram of Fig. 3, the lookup table memory 39 and the 

sensitivity pattern memory 40 are both physically implemented, for example, in the ROM 14, 
and the mapped pixel position brightness value memory 41 is physically implemented, for 
example, in the RAM 15. 

Turning back to Fig. 24, the mapper 42 receives from the captured image memory 20 a 
20 captured image brightness value at a pixel position received from the pixel position generator 43, 
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and the mapper 42 applies to the captured image brightness value lookup table data stored in the 
lookup table memory 39 for the sensitivity level corresponding to the received pixel position, as 
obtained from the sensitivity pattern memory 40, and derives a mapped brightness value at the 
received pixel position for storage in the mapped pixel position brightness value memory 41 . 
The mapped brightness value is the captured image brightness value compensated by the inverse 
of the response function of a light-sensing element having the sensitivity of level corresponding 
to the received pixel position. Then the pseudoinverse estimator 44 computes a brightness value 
at a off-grid position provided by the off-grid position generator 45 using the mapped brightness 
values in the mapped pixel position brightness value memory 41 . The computed off-grid 
brightness value at the off-grid position is then stored in an off-grid brightness value storage 36. 

Referring to Fig. 25, there is shown a flow diagram 2500 for a computation process that 
may be carried out by the off-grid estimator 2400 of the diagram of Fig. 24. First, loop 215 is 
repeated for each pixel position (x,y) of the captured image from (0.5,0.5) to 
(xSize-0.5, ySize-0.5), as received from the pixel position generator 43 in Fig. 24. In loop 215 
steps 216 and 217 are performed. In step 216, lookup table data LUTj for the i-th sensitivity 
level is chosen by referring to the sensitivity level / corresponding to received pixel position 
(x,y). The sensitivity level corresponding to received pixel position (x,y) is obtained from the 
sensitivity pattern data S(x,y) stored in the sensitivity pattern memory 40 of Fig. 24. In step 217, 
the lookup table data LUTi is applied to a captured image brightness value Ic (x,y) at received 
pixel position (x,y) from the captured image memory 20. The brightness value compensated by 
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the inverse of the response function of a light-sensing element having sensitivity level i derived 
from the lookup table data LUT^ is referred to as the mapped brightness value, Iin(x,y), at 
received pixel position (x^y). After loop 215 is repeated for every pixel position as received from 
the pixel position generator 43 in Fig. 24, loop 21 8 is repeated for each off-grid position (x',y ') 
from (1,1) to (xSize- 1, ySize- 1), as received from the off-grid position generator 45 in Fig. 24. 
In loop 218, step 219, loop 220 and step 223 are carried out. In step 219 a value s and a value w 
are each initiahzed to 0. After step 219, loop 220 is repeated for four pixel positions (u,v) which 
are the four nearest neighbor pixel positions to the received off-grid position (x',y'). In loop 220, 
steps 221 and 222 are performed. In step 221 a mapped brightness value lj}i,y) at pixel position 
(u,v), is compared with a low threshold value 6l and a high threshold value 0h . If Ini(u,v) is 
greater than 0l and less than Oh, step 222 is performed. Otherwise, step 222 is skipped and the 
process moves to the next repetition of loop 218. The threshold values 0l and 6h are 
predetermined constant values, which generally represent the mapped noise level of a light- 
sensing element having the sensitivity level corresponding to pixel position (u,v), and the 
saturation or near saturation brightness value of a light-sensing element having the same 
sensitivity level, respectively. In step 222 a mapped brightness value Ini(u,v), which is greater 
than 0L and less than 0h, is added to s and 1 is added to w. After loop 220 is repeated for each 
nearest neighbor pixel position, step 223 is performed. In step 223, the brightness value B(x',y') 
at received off-grid position (x',y') is computed by dividing the value s by the value w. After 
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loop 218 is repeated for each off-grid position received from the off-grid position generator 45 in 
Fig. 24, the computation process represented by the flow diagram 2500 terminates. 

Referring to Fig. 26, there is shown a flow diagram 2600 of an alternative computation 
process which may be carried out by the pseudoinverse estimator 44 of the diagram of Fig. 24. 
5 The flow diagram 2600 of Fig. 26, which implements a 4 x 4 cubic pseudoinverse computation, 
can replace the portion of the flow diagram 2500 of Fig. 25 that includes step 219, loop 220 and 
step 223. Referring to the flow diagram 2600 of Fig. 26, in step 224 a value s and a value w are 
^ each initialized to 0. Then loop 225 is repeated for sixteen pixel positions (u,v) which are the 
Ljl sixteen nearest neighbor pixel positions to an off-grid position (x',y') received from the off-grid 
i^H) position generator 45 in Fig. 24. In loop 225, steps 226, 227 and 228 are performed. In step 226, 
I a mapped brightness value (u,v) at nearest neighbor pixel position (u,v) is compared with a 

Q low threshold value 6l and with a high threshold value 6h. If In,(u,v) is greater than 0l and less 
rU than 6h, step 227 is performed. Otherwise, steps 227 and 228 are skipped and the process moves 
y to the next iteration of the loop 225. The threshold values 0l and 6h are predetermined constant 
1 5 values, generally representing the mapped noise level of a light-sensing element having the 
sensitivity level corresponding to pixel position (u,v), and the saturation or near saturation 
brightness value of a light-sensing element having the same sensitivity level, respectively. In 
step 227 index values i and j are computed from the received off-grid position (x',y') and the 
nearest neighbor pixel position (u,v), where i and j are used to refer to values of a cubic 
20 interpolation kernel which is shown in Fig. 27. In step 228 s and w are each updated in 
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accordance with the equations shown in the block representing step 228. After loop 225 is 
repeated for each of the sixteen nearest neighbor pixel positions, step 229 is performed. In step 
229, a brightness value B at received off-grid position (x',y') is computed as the value s divided 
by the value w. The computation process represented by flow diagram 2600 is repeated for all 
off-grid positions received from the off-grid position generator 45 in Fig. 24 from x'+l to 
xSize- 1 and y'+l 1 to ySize- 1. 

Turning to Fig. 28, there is shown a data flow diagram 2800 of the on-grid resampler 23 
in the diagram of Fig. 2 according to an exemplary embodiment of the present invention. The 
on-grid resampler 2800 includes a pixel position generator 38 for sequentially providing pixel 
positions from (2.5,2.5) to (xSize-2.5, ySize-2.5), and an interpolator 37 which carries out a 
cubic interpolation operation that computes an interpolated on-grid brightness value at a pixel 
position received from the pixel position generator 38 from the estimated off-grid brightness 
values at 4 X 4 off-grid positions that are nearest neighbors to the pixel position received from 
the pixel position generator 38. This is illustrated in Fig. 29 which shows a diagram 2900 of a 
local area of pixel positions, which are represented as rectangles, and off-grid positions, which 
are represented by black circles. When the brightness value at a received pixel position (x,y) is 
computed, the 4 x 4 cubic interpolation kernel is multiplied by 4 x 4 off-grid brightness values 
within the local area from (x-2,y-2) to (x+2,y+2), which is shown as the shaded rectangle in the 
diagram of Fig. 29. An example of a 4 x 4 cubic interpolation kernel 2700 is shown in Fig. 27, 
The pixel position generator 38 in Fig. 28 sequentially generates pixel positions from (2.5, 2.5) to 
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(xSize--2.5, ySize-2.5) and provides the pixel positions to the interpolator 37. In this manner, 
resampled brightness values for an image having dimensions of (xSize-4,ySize-4) are generated 
and stored in the output image memory 21 in Fig. 28. 

Referring to Fig, 30, there is shovra a flow^ diagram 3000 describing computation process 
that may be performed by the interpolator 37 in Fig. 28. The computation process comprises 
loop 213, which includes step 214. Loop 213 is repeated for each pixel position (x, y) of the 
output image from (2.5, 2.5) to (xSize-2.5, ySize-2.,5), as received from the pixel position 
generator 38 in Fig. 28, where x and y are the coordinates of the pixel position and xSize and 
ySize are the dimensions of the captured image in the x and y directions, respectively. In step 
214 a brightness value of the output image Io(x-2, y-2) at pixel position (x-2, y-2) is derived by 
computing the product of the brightness values at 4 x 4 nearest neighbor off-grid positions to the 
received pixel position at (x,y), and a 4 x 4 cubic interpolation kernel (|), as shown by the 
equation in the block representing step 214 in the flow diagram 3000. The computed value is 
stored as the brightness value at the corresponding pixel position of the output image Io(x-2,y-2) 
in the output image memory 21 of the diagram in Fig. 28. After loop 213 is repeated for each 
pixel position received from the pixel position generator 38 in Fig. 28, the computational process 
represented by the flow diagram 3000 terminates. 

While the interpolator 37 of the diagram in Fig. 28 is described as performing a cubic 
interpolation, it is to be understood that any interpolation process which computes an 
interpolated value from two dimensional uniform grid data may be substituted for the cubic 
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interpolation. For example, other cubic kernels, Gaussian kernels, bi-linear interpolation or 
larger interpolation kernels can be used. Furthermore, shifting the pixel positions to coincide 
with the off-grid positions and taking the brightness values at the off-grid positions as the 
brightness values at respective coincident pixel positions can be used to derive resampled 
brightness values at the pixel positions. Examples of bi-cubic B-spline 3100, bi-linear 3200, 
6x6 two dimensional Gaussian 3300 and a simple shift filter kernel 3400 are shown in Figs. 31, 
32, 33 and 34, respectively. Each of these filter kernels may be substituted as (l)y in the flow 
diagram 3000 of Fig. 30. It is noted that the range of values for i and j in the flow diagram 3000 
of Fig. 30 changes according to the size of the filter kernel. For example, when the 6 x 6 
Gaussian filter 3300 of Fig. 33 is used, both i and j take on values from 0 to 5. 

In the embodiments of the present invention described thus far, the light-sensing elements 
of the image sensor have four different sensitivity levels. It will be understood that the present 
invention may be readily applied to embodiments where the light-sensing elements of the image 
sensor have a different number of sensitivity levels with only minor alterations to the four 
sensitivity level embodiments described hereinabove. For example, the light-sensing elements of 
the image sensor may have only two sensitivity levels, in which case the modifications to the 
embodiments are now described. 

In the image sensing part where the spatially varying sensitivity pattern is defined by a 
mask, the mask will have cells with two different transparencies, namely a dark cell and a bright 
cell. For example, the mask may have a checkerboard pattern in which adjacent cells in a row 
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and adjacent cells in a column will have different ones of the two transparencies. An example of 
such a mask 3500 is depicted in Fig. 35, in which bright cells are designated by reference 
numeral 148 and dark cells are designated by reference numeral 149. 

In the image processing part, the off-grid position generators 32 in the diagram of Fig. 16 
and 45 in the diagram of Fig. 24 may generate the same off-grid positions as defined in Fig. 15 
for the two sensitivity level case. However, alternative off-grid positions may be defined when 
there are only two sensitivity levels. 

Referring to Fig. 36, there is shovm a diagram 3600 of an alternative off-grid position 
pattern for the two sensitivity level case. In Fig. 36, pixel positions are represented by squares 
150 and off-grid positions are represented by black circles 151. Here the off-grid positions 151 
are again located at respective interstices of the pixel position array. However, the interstices in 
the diagram of Fig. 36 are located at respective intersections of the pixel rows 267 and the 
intermediate columns 270, and at respective intersections of the pixel columns 268 and the 
intermediate rows 269. As in the diagram of Fig. 15, the pixel rows 267 and the pixel columns 
268 are regularly spaced and mutually orthogonal, and each intermediate row 269 is parallel to 
the pixel rows 267 and extends medially between a respective adjacent pair of pixel rows 267, 
while each intermediate column 270 is parallel to the pixel colximns 268 and extends medially 
between a respective pair of pixel columns 268. It may be noted in the diagram of Fig. 36 that 
lines of adjacent off-grid positions are rotated by 45 degrees with respect to lines of adjacent 
pixel positions. 
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Turning to Fig. 37, there is shown a data flow diagram 3700 of the embodiment of the 
off-grid estimator of Fig. 16 modified for two sensitivity levels. The differences from the four 
sensitivity level case are in the number of resamplers used, in the number of weighting factor 
memories and in the number of multipliers. The position generator 32 is modified to generate 
the off-grid positions defined in Fig. 36. The sensitivity pattern memory 34 stores a 
predetermined spatially varying sensitivity pattern specifying one of the two sensitivity levels 
corresponding to each pixel position. 

Referring to Fig. 38, there is shown a flow diagram 3800 of the computation process that 
may be performed by the off-grid estimator represented by the data flow diagram 3700 of Fig. 
37. First, loop 230 is repeated for each off-grid position (x, y) from (0.5,1) to (xSize-0.5, 
ySize- 1), where x and y are coordinates of the off-grid position and xSize and ySize are the 
dimensions of the captured image in the x direction and the y direction, respectively. With 
reference to the diagram 3600 of Fig. 36, the off-grid positions for which Loop 230 is repeated 
are located at respective intersections of the intermediate rows 269 and the pixel columns 268. 
In loop 230, step 231, loop 232 and step 235 are carried out. In step 231 a value s is initialized to 
0. In loop 232, steps 233 and 234 are repeated for each i-th sensitivity level. In the present 
embodiment the number of sensitivity levels is two. In step 233 a sensitivity level off-grid 
brightness value p^ for the i-th sensitivity level at an off-grid position (x,y) received from the off- 
grid position generator 32 in Fig. 37 is computed from the captured image brightness values at 
respective pixel positions of the captured image I, by the resampling function Resamplei(I„S,x,y) 
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of the i-th resampler, where S represents the sensitivity pattern data from the sensitivity pattern 
memory 34 in Fig. 37. The computation of this function is the same as in the four sensitivity 
level case. In step 234 the sensitivity level off-grid brightness value pj for the i-th sensitivity 
level is multiplied by a predetermined weighting factor Wj for the same sensitivity level, and the 
5 product is added to s. After loop 232 is repeated for the two sensitivity levels, step 235 is 

performed. In step 235, brightness value B at the off-grid position (x,y) received from the off- 
grid position generator 32 in Fig. 37 is computed by applying the function Map(LUT,s) using the 
Q mapper 33 and associated lookup table memory 35 in Fig. 37. The application of this function is 

the same as in the four sensitivity level case. After step 235 is completed, the current repetition 
40 of loop 230 is completed. After loop 230 is repeated for all off-grid positions located at 
i,g respective intersections of the intermediate rows 269 and the pixel columns 268, the computation 
Q process proceeds to loop 236. The computation process of loop 236 is identical to that of loop 
J J=J 230 except that loop 236 is repeated for each off-grid position received from this off-grid 
5 position generator 32 in Fig. 37 from (1 ,0.5) to (xSize- 1 ,ySize- 0.5). Referring again to the 
15 dia gram 3600 of Fig. 36, the off-grid positions for which loop 236 is repeated are located at 

respective intersections of the pixel rows 267 and the intermediate columns 270. After loop 236 
is repeated for all such off-grid positions, the computation process represented by the flow 
diagram 3800 terminates. 

The embodiment of the off-grid estimator represented by the data flow diagram 2400 in 
20 Fig. 24 is also readily modified for two sensitivity levels. Turning to Fig. 39, there is shown a 
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flow diagram 3900 describing a computation process that may be carried out by the off-grid 
estimator represented by data flow diagram 2400 of Fig. 24 for the two sensitivity level case. 
The first loop 256 is repeated for each pixel position (x, y) of the captured image received from 
the pixel position generator 43 in Fig. 24 from (0.5, 0.5) to (xSize-0.5, ySize-0.5). In loop 256, 
steps 257 and 258 are performed. In step 257 lookup table data LUT^ for the i-th sensitivity level 
is selected by referring to the sensitivity pattern data S(x, y) stored in the sensitivity pattern 
memory 40 for the sensitivity level corresponding to pixel position (x, y) received from the pixel 
position generator 43. In step 258 the lookup table data LUTj is applied to the captured image 
brightness value Ic(x, y) at received pixel position (x, y), and the resulting compensated 
brightness value at received pixel position (x, y) is designated the mapped brightness value l^(x, 
y) at that pixel position. After loop 256 is repeated for all pixel positions of the captured image 
received from pixel position generator 43, the computation process proceeds to loop 259. Loop 

259 is repeated for each off-grid position (x\y') from (0.5, 1) to (xSize-0.5, ySize-1) received 
from the off-grid position generator 45 in Fig. 24. With reference to the diagram 3600 in Fig. 36, 
the off-grid positions for which loop 259 is repeated are located at respective intersections of the 
intermediate rows 269 and the pixel columns 268. In loop 259, step 260 is carried out. In step 

260 a function Pseudoinverse(In,, x', y') is evaluated to obtain the brightness value B(x', y') at 
received off-grid position (x', y'). The details of evaluating the fimction Pseudoinverse (I^, x', y") 
will be fiirther explained hereinbelow. After loop 259 is repeated for each off-grid position 
located at respective intersections of the intermediate rows 269 and the pixel columns 268, as 
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received from off-grid position generator 45, the computation process proceeds to loop 261, 
which is repeated for each off-grid position from (1, 0.5) to (xSize-1, ySize-0.5). With reference 
to diagram 3600 of Fig. 36, the off-grid positions for which loop 261 is repeated are located at 
respective intersections of the pixel rows 267 and the intermediate columns 270. Otherwise, the 
computation process of loop 261 is identical to that of loop 259. After loop 259 is repeated for 
each of the off-grid positions located at respective intersections of the pixel rows 267 and the 
intermediate columns 270, as received from off-grid position generator 45, the computation 
process represented by flow diagram 3900 terminates. 

Referring to Fig. 40, there is shown a flow diagram 4000 describing the computation 
process for the function Pseudoinverse(Iin, ^\ y% which carries out a 4 x 4 rotated cubic 
pseudoinverse operation. First, in step 263 a value s and a value w are each initialized to 0. 
After step 263, loop 264 is repeated for 16 pixel positions (u, v) which are nearest neighbors to 
an off-grid position (x', y') received from the off-grid position generator 45 by incrementing 
indices i and j, which have values from 0 to 3, in each repetition of loop 264. In loop 264, steps 
265, 266, 267 and 268 are carried out. In step 265, coordinates u and v are computed from the 
coordinates of the current received off-grid position (x, y') and indices i and j using the relations 
shown in the block representing step 265 in the flow diagram 4000. In step 266 a determination 
is made as to whether the coordinates u and v are those of a pixel position. If coordinates u and v 
are those of a pixel position, the mapped brightness value I^(u, v) at that pixel position is 
compared with a low threshold value 0l and a high threshold value Oh in step 267. Otherwise, 
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steps 267 and 268 are skipped and the computation process moves to the next repetition of loop 
264. If in step 267 the mapped brightness value lju, v) is determined to be greater than the low 
threshold value and less than the high threshold value 6h, then step 268 is performed. 
Otherwise, step 268 is skipped and the computation process moves to the next repetition of loop 
264. The threshold values 6^ and 6h are predetermined constants, which generally represent the 
mapped noise threshold of a light-sensing element of the image sensor having the sensitivity 
level corresponding to pixel position (u,v), and the saturation or near saturation brightness value 
of a light-sensing element of the image sensor having the same sensitivity level, respectively. In 
step 268, the values s and w are updated in the accordance with the equations shown in the block 
representing step 268 in the flow diagram 4000. In these equations, c^^^i^epresents the value of a 
45 ° rotated cubic interpolation kernel <^ at indices m and n. An example of a 45 ° rotated cubic 
interpolation kernel is shown in diagram 4200 in Fig. 42. After loop 264 is repeated for all 
values of the indices i and j, step 269 is performed. In step 269 the value of the function 
Resample(I^, x', y') is obtained by dividing the value s by the value w, and the computation 
process represented by the flow diagram 4000 is repeated for the next off-grid position (x\y ') 
received from the off-grid position generator 45 in Fig. 24. 

The data flow diagram of the on-grid resampler for the two sensitivity level case is the 
same as that for the four sensitivity level cases, as shown in Fig. 28, However, there are some 
differences in the computation carried out by the interpolator 37 in the diagram of Fig. 28. As in 
the four sensitivity level case, the interpolator carries out a cubic interpolation operation which 
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computes an interpolated brightness value at a pixel position received from pixel position 
generator 38 in Fig. 28 using respective estimated off-grid brightness values at 4 x 4 off-grid 
positions that are nearest neighbors to the received pixel position. The shape of the interpolation 
filter kernel for two sensitivity levels is, however, different. Figure 41 shows a diagram of a 
local area of pixel positions represented by squares and off-grid positions represented by black 
circles. When the brightness value at a received pixel position (x, y) is computed, a 45^ rotated 4 
X 4 cubic interpolation kernel is applied to form a product with the brightness values at 4 x 4 off- 
grid positions that are nearest neighbors to the received pixel position (x,y) within a diamond 
shaped area from (x-2, y-2) to (x+2, y+2), which is shown as a shaded diamond shaped region in 
Fig. 41 . An example of a 45 rotated 4x4 cubic interpolation kernel 4200 is shown in Fig. 42. 
Since the filter kernel for two sensitivity levels is more compact than the kernel for four 
sensitivity levels, the embodiment for two sensitivity levels results in better spatial frequency 
characteristics. 

Turning to Fig. 43, there is shown a flow diagram 4300 describing a computation process 
that may be carried out by the embodiment of the on-grid resampler for two sensitivity levels. 
The computation process comprises loop 242, which includes step 243. Loop 242 is repeated for 
each pixel position (x, y) of the output image received from pixel position generator 38 in Fig. 28 
from position (2.5, 2.5) to position (xSize-2.5, ySize-2.5), where x and y are the coordinates of 
the pixel position, and xSize and ySize are the dimensions of the captured image in the x 
direction and the y direction, respectively. In step 243, a brightness value at a received pixel 
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position (x,y) of the output image Io(x,y) is derived by computing the product of the brightness 
value B at the 4 X 4 off-grid positions that are nearest neighbors to the received pixel position 
(xj% and the 45 rotated 4x4 cubic interpolation kernel (j) in accordance with the equation 
shown in the block representing step 243 in the flow diagram 4300. The computed brightness 
5 value at the corresponding pixel position of the output image I (x-2,y-2) is stored in the output 
image memory 21 in Fig. 28. After repetition of loop 242 for every pixel position received from 
pixel position generator 38 in Fig. 28 is completed, the computation process represented by the 
□ flow diagram 4300 terminates. 

In Referring back to the data flow diagram 200 of Fig. 2, it is noted that where the o&grid 

"IJO estimator 22 derives respective off-grid brightness values at off-grid positions that lie in a regular 
■ J:^ array such as that shown in Fig. 15 and Fig. 36, the regular array of off-grid brightness values 
may be directly stored in the output image memory 21 as the pixel brightness values of a high 
i y dynamic range output image, without further processing by an on-grid resampler 23. This is 

shown in the exemplary embodiment of the present invention represented by the data flow 
15 diagram of Fig. 44, where the respective estimated off-grid brightness values computed by the 
off-grid estimator 22 are provided directly to the output image memory 21 for subsequent 
display or further processing. Although an output image based directly on the estimated off-grid 
brightness values will generally not have as high a quality as one that has undergone resampling 
by the on-grid resampler 22, the quality may be sufficient for some applications. 
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Turning to Fig. 45, there is shown a data flow diagram 4500 according to another 
exemplary embodiment of the present invention, in which the mapper 33 and associated lookup 
table memory 35 of the off-grid estimator of Fig. 16 are moved from the location shown in Fig. 
16 to after the interpolator 37 of the on-grid resampler of Fig. 28. Approximately the same result 
is obtained using the arrangement of Fig. 45 as is obtained with the combination of the off-grid 
estimator of Fig. 16 and the on-grid resampler of Fig. 28. 

Turning to Fig. 46, there is shown a flow diagram 4600 of the computational process that 
may be carried out by the arrangement of Fig. 45. First, loop 244 is repeated for each off-grid 
position (x', y') received from off-grid position generator 32 in Fig. 45 from (1, 1) to 
(xSize-l,ySize-l), where x' and y' are the coordinates of the off-grid positions, and xSize and 
ySize are the dimensions of the captured image in the x direction and y direction, respectively. 
In loop 244, step 245, loop 246 and step 249 are carried out. In step 245 a value s is initialized to 
0. In loop 246, steps 247 and 248 are repeated for each i-th sensitivity level. In the present 
embodiment, the number of sensitivity levels is 4. In step 247 a sensitivity level off-grid 
brightness value p^ for the i-th sensitivity level at received off-grid position (x', y') is derived from 
the captured image brightness values 1^ in the captured image memory 20 by the resampling 
fimction Resamplei(Ic, S, x', y') of the i-th resampler, where S represents the sensitivity pattem 
data from the sensitivity pattem memory 34 in Fig. 45. In step 248 the computed sensitivity 
level off-grid brightness value for the i-th sensitivity level p^ is multiplied by a predetermined 
weighting factor Wi for the i-th sensitivity level The product of the multiplication is added to s. 
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After loop 246 is repeated for each sensitivity level, step 249 is performed. In step 249 the value 
s is assigned to the off-grid brightness value B at off-grid position (x , y') received from off-grid 
position generator 32 in Fig. 45. After step 249 is performed, the current repetition of loop 244 
is completed. After loop 244 is repeated for every off-grid position, as received from off-grid 
5 position generator 32, the computation proceeds to loop 250. Loop 250 is repeated for each pixel 
position (x, y) from (2.5, 2.5) to (xSize-2.5, ySize-2.5), as received from pixel position generator 
38 in Fig. 45. In loop 250, steps 251 and 252 are performed. In step 251 an interpolated on-grid 
:;:3 brightness value I at pixel position (x,y) is derived by computing the product of the brightness 
y I values at 4 X 4 off^grid positions that are nearest neighbors to a received pixel position (x, y) and 
'^JO a 4 X 4 cubic interpolation kemel using the equation in the block representing step 251 in the 
J flow diagram 4600. In step 252, the fimction MAP(LUT,I) is evaluated to obtain the pixel 
a brightness value of the output image Io(x-2, y-2) at pixel position (x-2,y-2). The computation 

process for evaluating the ftmction MAP(LUT,I) is described by the flow diagrm 1900 in Fig. 19. 
After loop 250 is repeated for every pixel position received from pixel position generator 38 in 
15 Fig. 45, the computation process represented by the flow diagram for 4600 terminates. 

Referring now to Fig. 47, there is shown a data flow diagram 4700 of another exemplary 
embodiment of the present invention, which uses an on-grid interpolation filter 46 to implement 
the image processing part 2 of Fig. 1. Comparing the diagram in Fig. 47 with the diagram in Fig. 
45, it is noted that the resamplers 28, 29, 30 and 31, the weighting factor memories 24, 25, 26 
20 and 27, the sensitivity pattern memory 34, the off-grid brightness value memory 36, the 
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interpolator 37, the off-grid position generator 32 and the pixel position generator 38 are replaced 
by an on-grid interpolation filter 46 and a pixel position generator 47. The derivation of the filter 
is now explained. 

In accordance with the present invention, off-grid brightness values B(x,y) are obtained 
by applying an off-grid estimating function^ff(*) to the captured image brightness values Ic(x,y), 
i.e.: 



In the embodiment of the off-grid estimator represented by Figs. 16-19, off-grid brightness 
values dependent on a monotonic light-sensing element response function s(E) are first derived, 
and then response function independent off-grid brightness values are recovered by applying a 
lookup table for the inverse of the response function s(E), i.e.: 



where ^'(*) is a function for obtaining sensitivity level dependent off-grid brightness values by 
using a weighted sum of response functions of light-sensing elements having different sensitivity 
levels such as that computed using equation (1). 

To obtain the output image, an on-grid resampling functiony^nC*) is used to compute 
brightness values at pixel positions by a conventional resampling function r(*). Therefore, the 
output image can be expressed as 



B(x,y) = f,/Ii:x,y)). 



(2) 



(3) 
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= r(s-'(s(I/x,y))). 

It may be assumed that the same continuous function can be constructed from different 
resampled data" sequences if the original continuous function is sufficiently smooth. Based on 
this assumption, /(xj^)is equivalent to lo(x,y), where 

= s-\ris(Ilx,y))y (5) 

Although and are not equivalent when s(IJxy))is. not smooth, / (xj^)can nevertheless be 
used as an approximation of I(j(x,y). 

The functions ^(*) and r(*) may be combined into a single filter function because both 
functions are linear and invariant to phase shift. Therefore the embodiment represented by the 
diagram of Fig. 47 uses an on-grid interpolation filter that combines s(^) and r^*"). The filter is 
derived by the convolution of s(^) and r(*) . An example of the kernel 4900 of such a filter that 
combines s(^) and r(*)is shown in Fig. 49. The filter kernel 4800 is derived by using simple 
averaging of four nearest neighbor pixel position brightness value resampling described for the 
embodiment of the off-grid estimator of Figs, 16-19 as ^'(*), and conventional cubic 
interpolation filtering as r(*). 

Referring to Fig, 48, there is shown a flow diagram 4800 describing the computation 
process carried out by the embodiment of the image processing part represented by the data flow 
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diagram 4700 in Fig. 47. First, loop 253 is repeated for each pixel position (x, y) of the output 
image, as received from pixel position generator 47 in Fig. 47, from (2.5,2.5) to 
(xSize-2.5,ySize-2.5), where x and y are the coordinates of the pixel position, and xSize and 
ySize are the dimensions of the captured image in the x and y directions, respectively. In loop 
253, steps 254 and 255 are carried out. In step 254 an interpolated on-grid brightness value I is 
computed by multiplying the brightness values of the captured image 1^ at 5 x 5 nearest neighbor 
pixel positions to a received pixel position (x,y) by a 5 x 5 filter kernel (|). In step 255 the lookup 
table data, which is stored in the lookup table memory 35 in Fig. 47, is applied by the mapper 33 
in Fig. 47 to the computed on-grid brightness value I to obtain the brightness value of the output 
image Io(x-2,y-2) at pixel position (x-2,y-2). After loop 253 is repeated for every output 
image pixel position received from the pixel position generator 47 in Fig. 47, the computation 
process represented by the flow diagram 4800 terminates. Using the process represented by flow 
diagram 4800, an output image with dimensions of xSize-4 and ySize-4 is obtained. 

The present invention may be applied to linear image sensors having linear arrays of 
light-sensing elements that capture line images. Such linear image sensors, including CCD linear 
image sensors, are available with a greater number of light-sensing elements along a given 
direction than two-dimensional image sensors. Accordingly, linear image sensors may be used 
to provide two-dimensional images of greater resolution in applications where scaiming of a 
scene with the linear image sensor is feasible and sequentially captured line images of successive 
regularly spaced linear regions of the scene are concatenated. Examples of the ways that a linear 
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image sensor may be used to scan a scene and sequentially capture line images that may be 
concatenated to for a two-dimension image of the scene are illustrated in Figs. 50 and 5 1 . 
Referring to Fig. 50, there is shown an imaging system having a linear image sensor 276 and a 
lens system 277 for projecting a line image of a linear region 310 of a scene having an object 278 
5 onto the light-sensing elements (not shown) of the linear image sensor. The imaging system is 
moved (translated and/or rotated) at a constant rate with respect to the scene as shown by the 
arrow indicating the motion of the imaging system in the figvire. Line images of the scene are 
Q captured at regular sequential intervals by applying a regular sequence of exposure signals (not 
in shown) to the linear image sensor 276 dviring the motion. The sequentially captured line images 

' |0 can be concatenated to form a single high resolution two-dimensional image of the scene. 
: n Turning to Fig. 51, there is shown an imaging system comprising a linear image sensor 

i:.3 276 and a lens system 277. The imaging system is held stationary with respect to a scene 
PU comprising a conveyor beh 279 carrying objects 280 and 281 moving past the imaging system at 
a constant speed. Line images of successive regularly spaced linear regions 3 1 1 of the scene are 
15 sequentially captured at regular intervals by applying a regular sequence of exposure signals (not 
shown) to the linear image sensor 276. The sequentially captured line images can again be 
concatenated to form a two-dimensional image of a section of the conveyor belt 279 carrying 
objects 280 and 281. 

In accordance with the present invention, the linear image sensor 276 has a spatially 
20 varying sensitivity pattern. Referring to Fig. 52, there is shown an illustrative linear mask 5200 
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having cells 282 and 283 with two different transparencies. Each cell of the mask 5200 is placed 
in front of a respective one of the light-sensing elements of the linear image sensor so as to cause 
each light-sensing element of the sensor to have a respective one of two sensitivity levels in 
accordance with the predetermined sensitivity pattern of the linear array of light-sensing 
5 elements as defined by the mask 5200. 

Each light-sensing element of the linear image sensor in response to incident light from 
the scene produces a captured image brightness value at a corresponding one of a multiplicity of 
r-i regularly spaced pixel positions of a linear pixel position array, thereby capturing a line image. 
iJi When successive line images captured by the linear image sensor are concatenated, there is 

^'f 0 obtained a two-dimensional array of pixel positions located at respective intersections of a 

plxirality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns 
i;3 orthogonal to the pixel rows, where each pixel position has a respective brightness value 

ilJ produced by a light-sensing element that corresponds to the same pixel position in each one of 

the concatenated line images, and corresponds to a respective one of the two sensitivity levels of 
15 the light-sensing elements in accordance with a two-dimensional sensitivity pattern as defined 
illulstratively by the concatenated exposure mask of Fig. 53. It will be understood that the light- 
sensing elements of the linear image sensor may have more than two sensitivity levels and that 
the sensitivity level of each light-sensing element may be established by means other than a 
mask. 
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Referring back to Fig. 37, the captured image brightness values at respective pixel 
positions of the concatenated two-dimensional captured image may be stored in captured image 
memory 20 and the spatially varying sensitivity pattern as represented illustratively by Fig. 53 
may be stored in the sensitivity pattern memory 34. The off-grid brightness values may be 
5 estimated in accordance with the data flow diagram shown in the figure, and the array of off-grid 
positions may be a concatenation of the linear off-grid position array shown illustratively in Fig. 
54, where each off-grid position 286 is located midway between a respective adjacent pair of 
r 3 pixel positions 285. 

Ul The data flow diagram of Fig. 24 and the computational processes described by the flow 

^'JO diagrams 3900 and 4000 of Figs. 25 and 30 with appropriate modifications to account for a 

different off-grid position array may also be used to estimate the off-grid position brightness 
\n values from the concatenated two-dimensional captured image brightness values in the captured 
i U image memory 20. The estimated off-grid brightness values at the respective off-grid positions 
^::f may be used directly as the pixel brightness values of an output image in accordance with Fig. 
15 44, or on-grid resampUng of the estimated off-grid brightness values in accordance with the data 
flow diagram of Fig. 28 and the computation process described by the flow diagram 3000 of Fig. 
30 with appropriate modifications to account for a different off-grid array, may be used to derive 
the resampled on-grid brightness values at respective pixel positions of the output image, the 
resampled on-grid brightness values being the pixel brightness values of the output image stored 
20 in the output image memory 2 1 . 
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Referring to Figs. 45 and 46, the captured image brightness values at respective pixel 
positions of the concatenated captured line images stored in the captured image memory 20 may 
be processed in accordance with the data flow diagram 4500 of Fig, 45 using the computation 
process described by the flow diagram 4600 of Fig. 46 to derive resampled brightness values at 
5 respective pixel positions of the output image to be stored in the output image memory 21 . To 
carry out such processing, the data flow diagram 4500 and the computation flow diagram 4600 
are modified to take into account the use of an image sensor with light-sensing elements having 
Q two sensitivity levels and the use of a different off-grid position array. 

Referring to Figs. 47 and 48, the captured image brightness values at respective pixel 
Jo positions of the concatenated captured line images stored in the captured image memory 20 may 
: □ be processed in accordance with the data flow diagram 4700 of Fig. 47 using the computation 

Q process described by the flow diagram 4800 of Fig. 48 to derive compensated interpolated 
\ ^ brightness values at respective pixel positions of the output image to be stored in the output 
ir5 image memory 21 . To carry out such processing, the data flow diagram 4700 and the flow 
1 5 diagram 4800 are modified to take into account the use of an image sensor with light-sensing 

elements having two sensitivity levels. Aside from changes in the data stored in the lookup table 
35 representing the inverse of a combined response fimction of light-sensing elements having the 
two sensitivity levels, these modifications will also result in a different filter kemel (|) to be used 
in the computation process described by the flow diagram 4800. 
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As an alternative to processing concatenated captured line images, each line image may 
be separately processed to derive a high dynamic range line image by estimating the brightness 
values at off-grid positions, which for a linear pixel position array are located between respective 
adjacent pairs of pixel positions. The processed line images may then be concatenated to obtain 
5 a high dynamic range two-dimensional image. Separate processing of each line image is less 
computationally intensive in comparison to the processing of a two-dimensional image, and 
therefore can be done in real time as the line images are captured. Turning to Fig. 54, there is 
u shown an illustrative row of pixel positions 285 represented by squares, and "off-grid" positions 
i^'j 286 represented by circles, where each off-grid position 285 is located midway between a 
Jo respective adjacent pair of pixel positions 285. According to one exemplary embodiment, the 
: g brightness value at each one of the off-grid positions 286 may be derived by computing a 
Q weighted average of the captured image brightness values at the pixel positions 285 within a pre- 
\ ^ defined neighborhood of the off-grid position. In the case where the linear image sensor has 
S light-sensing elements of two different sensitivity levels, the captured image brightness values at 
15 all consecutive adjacent pairs of pixel positions are averaged to obtain the estimated brightness 
value at the off-grid position between each of the consecutive adjacent pairs of pixel positions. 
In this manner, the brightness value B(x') at off-grid position x' may be expressed as 

B(x') = W^ip' - 0.5) + W^I^ix' + 0.5), (6) 
where Wj and W2 are weighting factors assigned to the two different sensitivity levels. The 
20 resulting off-grid brightness values represent a one-dimensional image with high dynamic range. 
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The computed off-grid brightness values are each compensated by the inverse of a combined 
response function of the Kght-sensing elements. The combined response function may be 
expressed as 

s(E)-W^P^(E) W/^(E), (7) 
where Pi(E) is the radiometric response function of a light-sensing element of the linear image 
sensor having one of the two sensitivity levels, P2(E) is the radiometric response function of a 
light-sensing element of the linear image sensor having the other of the two sensitivity levels, 
and Wi and W2 are the weighting factors assigned to the two different sensitivity levels . 
Compensation of the estimated off-grid brightness values by the inverse of the combined 
response function of the light-sensing elements using a mapper and a lookup table memory as in 
the two-dimensional case remaps the off-grid brightness values to ones that are linearly related to 
scene radiance. 

Referring to Fig. 55, there is shown a flow diagram 5500 describing a computation 
process for deriving the estimated brightness values at the off-grid positions 286 in Fig. 54. The 
computation process described by the flow diagram 5500 is carried out by an off-grid estimator 
represented by the data flow diagram 2400 of Fig. 24. Referring to Fig. 24, the pixel position 
generator 43 provides the positions of the pixel position 285 of the linear pixel position array 
shown in Fig. 54. The sensitivity pattern memory 40 stores the sensitivity level corresponding to 
each pixel position as defined by the illustrative mask 5200 in Fig. 52. The lookup table memory 
39 stores data representing respective inverse response function data of light-sensing elements 
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having the two sensitivity levels. The off-grid position generator 46 provides the off-grid 
positions 286 of the off-grid position array shown in Fig. 54. 

Turning back to Fig. 55, loop 287 is first repeated for each pixel position x of the linear 
pixel position array from 0.5 to xSize-0.5 as received from the pixel position generator 43 in 
Fig. 24, where xSize is the length of the line image. In loop 287, steps 288 and 289 are 
performed. In step 288, lookup table data LUTj for the i-th sensitivity level is chosen by 
referring to the sensitivity level i corresponding to received pixel position x. The sensitivity 
level corresponding to the received pixel position x is obtained from sensitivity pattern data S(x) 
stored in the sensitivity pattern memory 40 in Fig. 24. In step 289, the lookup table data LUTj 
stored in the lookup table memory 39 in Fig. 24 is applied by the mapper 42 in Fig. 24 to the 
captured image brightness value at received pixel position x from the captured image 
memory 20 in Fig. 24. The brightness value at received pixel position x compensated by the 
inverse of the response function of the light-sensing element having sensitivity level i derived 
from the lookup table data LUTj is designated by Iin(x). After loop 287 is repeated for every 
pixel position received from the pixel position generator 43 in Fig. 24, loop 290 is repeated for 
each off-grid position x' from 2 to xSize-2 as received from the off-grid position generator 45 in 
Fig. 24. In loop 290, step 291, loop 292 and step 293 are performed. In step 291, a value p and a 
value q are each initialized to 0. After step 291, loop 292 is repeated for each pixel position k 
that is one of a predetermined number of pixel positions that are nearest neighbors to the received 
off-grid position x'. In the computation process described by the flow diagram 5500 of Fig. 55, 
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the predetermined number of nearest neighbor pixel positions is four and k has values from 
(x'- 1 .5) to (x'+l ,5). In loop 292, steps 294 and 295 are performed. In step 294, a compensated 
captured image brightness value at pixel position k , IJk), is compared with a low threshold 
value 6l and a high threshold value 6h. If Im(k) is greater than 0l and less than 6^, step 295 is 
5 performed. Otherwise, loop 292 proceeds to its next repetition. In step 295, l^(k) is multiplied 
by the value of a 4 x IGaussian interpolation kernel G(k-x') at position (k-x')^ and the product 
is added to the quantity p. In addition, the value of the Gaussian interpolation kernel G(k-x') is 
Q added to the quantity q. After loop 292 is repeated for each value of k, step 293 is performed. In 
\M step 293, the off-grid brightness value B(x') at the received off-grid position x' is computed by 

- JO dividing the value p by the value q, and loop 290 proceeds to the next off-grid position received 

from the off-grid position generator 45 in Fig. 24. After loop 290 has been repeated for each off- 
rj grid position x' of the off-grid position array, the computation process described by flow diagram 
rU 5500 terminates. The threshold values 6^ and 9h are predetermined constant values, which 

generally represent the noise level of a light-sensing element having the sensitivity level 
1 5 corresponding to pixel position k compensated by the inverse of the response function of the 
light-sensing element, and the saturation or near saturation value of a light-sensing element 
having the same sensitivity level, respectively. An example of a 4 x 1 Gaussian kernel 5600 that 
may be used in the computation process described by the flow diagram 5500 is shown in Fig. 55, 
assuming that the center of the Gaussian kernel 5600 is at the center of the 4 x 1 pixel position 
20 array involved in the interpolation. 
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The respective estimated off-grid brightness values at the off-grid positions may be used 
directly as the pixel brightness values of an output image and stored in the output image memory 
36 in Fig. 24. Alternatively, the off-grid brightness values may be resampled using one of a 
variety of interpolation techniques to obtain resampled pixel position brightness values that are 
5 then stored in the output image memory as pixel brightness values of the output image. 

Referring back to Fig. 28, resampling of the estimated off-grid brightness values to derive 
respective resampled brightness values at the pixel positions may be in accordance with the data 
flow diagram of 2800. An interpolator 37 receives pixel positions of the linear pixel position 
Lil array illustrated in Fig. 54 from the pixel position generator 38. For each received pixel position 

0 the interpolator 37 derives from the estimated off-grid brightness values stored in the off-grid 
' brightness value memory 36 an interpolated brightness value at the received pixel position, 

Q which is then provided to the output image memory 21 . Turning to Fig. 57, there is shown a 
fy flow diagram 5700 describing the computation process that may be performed by the interpolator 
y 37 in Fig. 28. The computation process comprises loop 297, which includes step 298. Loop 297 
15 is repeated for each pixel position x of the output image from 3.5 to xSize~3.5, as received from 
the pixel position generator 38 in Fig. 28, where xSize is the length of the captured line image. 
In step 298 a brightness value of the output image Io(x-3) is derived by computing the product of 
the brightness values at four nearest neighbor off-grid positions to the received pixel position x, 
and a 4 X 1 cubic interpolation kernel The computed on-grid brightness value Io(x-3) at pixel 
20 position (x-3) is stored as the brightness value at the corresponding pixel position of the output 
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image in the output image memory 2 1 of Fig. 28. After loop 297 is repeated for each pixel 
position from 3.5 to xSize-3.5 received from the pixel position generator 38 in Fig. 28, the 
computation process described by the flow diagram 5700 terminates. An example of a cubic 
interpolation kernel \|j (k) 5800 that may be used in the computation process described by the 
flow diagram 5700 is shown in Fig. 58. 

Another embodiment of the present invention for processing captured line images to 
derive on-grid brightness values of a line output image is represented by the data flow diagram 
4700 of Fig. 47. Referring to Fig. 47, the captured image brightness values at respective pixel 
positions of a linear pixel position array as illustrated in Fig. 54 are stored in the captured image 
memory 20. An on-grid interpolation filter 46 receives pixel positions of the linear pixel position 
array from a pixel position generator 47. For each pixel position received from the pixel position 
generator 47 the filter 46 interpolates the captured image brightness values in the captured image 
memory 20 to derive an interpolated on-grid brightness value at the received pixel position. The 
interpolated on-grid brightness value is then compensated by the inverse of a combined response 
function of the light-sensing elements of the image sensor (not shovm), and the compensated 
interpolated on-grid brightness value is stored in the output image memory 21 as an output line 
image brightness value at the pixel position received from the pixel position generator 47. 

Referring to Fig. 59, there is shown a data flow diagram 5900 representing a fiarther 
embodiment of the image processing part 2 of the data flow diagram of Fig. 1 . When used for 
processing captured line images, the image processing part of Fig. 59 includes a first pixel 
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position generator 43 for sequentially generating pixel positions of the linear pixel position array 
as illustrated in Fig. 54 from 0.5 to xSize-0.5, where xSize is the length of the line image. The 
image processing part of Fig. 59 also includes a sensitivity pattern memory 40 for storing the 
sensitivity level corresponding to each pixel position of the linear pixel position array, a mapper 
5 42 and associated lookup table memory 39 for storing separate lookup table data representing the 
inverse of the response functions of light-sensing elements having different ones of the 
sensitivity levels, a mapped pixel position brightness value memory 41 for storing the captured 
ri image brightness value at each pixel position after compensation by the inverse of the response 
m function of a light-sensing element having a sensitivity level corresponding to the pixel position, 
^lO a second pixel position generator 47 for generating pixel positions of the pixel position array 
.1:=^ from 2.5 to xSize-2.5, an on-grid interpolation filter 46 receiving pixel positions from the second 
i;3 pixel position generator 43 and interpolating the compensated captured image brightness values 
i U at respective pixel positions in the mapped pixel position brightness value memory 41 to derive 

an on-grid interpolated brightness value for each pixel position received from the second pixel 
1 5 position generator 47. The on-grid interpolated brightness values at respective pixel positions of 
the linear pixel position array are stored in the output image memory 21 as the pixel brightness 
values of an output line image. It is noted that the embodiment of the image processing part 
represented by the data flow^ diagram 5900 of Fig. 59 may be used to process two-dimensional 
captured images with appropriate changes to take into account a two-dimensional pixel position 
20 array, a two-dimensional sensitivity pattern and a two-dimensional on-grid interpolation filter. 
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Turning to Fig. 60, there is shown a flow diagram 6000 describing a computation process 
that may be carried out by the embodiment of the image processing part represented by the data 
flow diagram 5900 of Fig. 59 for processing a captured line image. First, loop 299 is repeated 
for each pixel position x received from the first pixel position generator 43 in Fig. 59 from 0,5 to 
5 xSize-O.S, where xSize is the length of the captured line image. In loop 299, steps 301 and 302 
are performed. In step 301 lookup table data LUTj for the i-th sensitivity level stored in the 
lookup table memory 39 in Fig. 59 is selected by referring to the sensitivity pattem data S(x) 
n stored in the sensitivity pattem memory 40 in Fig. 59 for the sensitivity level corresponding to 
In pixel position x received from the first pixel position generator 43 in Fig. 59. In step 302, the 
='r0 lookup table data LUTj is applied to the captured image brightness value l^(x) at the received 

pixel position x by the mapper 42 in Fig. 59, and the resulting compensated captured image 
Q brightness value at received pixel position x is designated the mapped brightness value, Iin(x), at 
ry that pixel position. After loop 299 is repeated for all pixel positions of the pixel position array 
::!f received from the first pixel position generator 43, the computation process proceeds to loop 303. 
15 Loop 303 is repeated for each pixel position x from 2.5 to xSize-2.5. In loop 303, step 304, loop 
305 and step 306 are performed. In step 304, quantities p and q are each set to 0. After step 304, 
loop 305 is repeated for five pixel positions k which are nearest neighbors to the pixel position x 
received from the second pixel position generator 47 by incrementing the pixel position k in each 
repetition of loop 305 from k = x-2 to k = x+2. In loop 305, steps 307, 308 and 309 are carried 
20 out. In step 305, a mapped brightness value at pixel position k, Ij^(k), is compared with a low 
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threshold value 6l and a high threshold value 0h. If Im(k) is greater than Gl and less than 0h, step 
308 is performed. Otherwise, loop 305 proceeds to the next value of k. In step 308, is 
multiplied by the value of a linear Gaussian interpolation kernel G(k-x) at position (k-x), and 
the product is added to the quantity p. In addition, the value of the Gaussian interpolation kernel 
5 G(k-x) at position (k-x) is added to the quantity q. After loop 305 is repeated for each value of 
k from k=x-2 to k=x+2, step 306 is performed. In step 306, the output image brightness value 
Io(x) at the pixel position x received from the second pixel position generator 47 is computed by 
dividing the value p by the value q. The loop 303 then proceeds to the next pixel position 
yl received from the second pixel position generator 47. After loop 303 has been repeated for each 
' fO pixel position x from x=2.5 to x=xSize-2.5, the computation process represented by flow 

diagram 6000 terminates. The threshold values 6l and 0h are predetermined constant values, 
[1 which generally represent the noise level of a light-sensing element having the sensitivity level 
ry corresponding to pixel position k compensated by the inverse of the response function of that 

;::f light-sensing element, and the saturation or near saturation value of the light-sensing element 
15 having the same sensitivity level, respectively. An example of a 5 x 1 Gaussian kernel G(k), 

which may be used in the computation process described by the flow diagram 6000 is shown in 
Fig. 61. 

While the present invention has been particularly described with reference to exemplary 
embodiments thereof, it will be understood by those skilled in the art that various modifications 
20 and alterations may be made without departing of the spirit and the scope of the invention. For 
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example, the image sensor need not be a charge coupled device, but may be any image sensor 
having an array of light-sensing elements, such as a CMOS image sensor. Accordingly, the 
disclosed embodiments of the invention are considered merely ilUustrative, and the invention is 
limited in scope only as specified in the appended claims. 
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WE CLAIM : 

1 LA method for obtaining a relatively high dynamic range image of a scene using a 

2 relatively low dynamic range image sensor exposed to incident light from the scene for capturing 

3 an image thereof, the image sensor having a multiplicity of light-sensing elements in an array, 

4 each light-sensing element having a particular one of a plurality of sensitivity levels to incident 

5 light in accordance with a predetermined sensitivity pattern for the array of light-sensing 

6 elements and having a response function, each light-sensing element being responsive to incident 
;.p7 light from the scene for producing a captured image brightness value at a corresponding one of a 
i^^^8 multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity of 
;'u9 pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light- 
■'TO sensing elements, the method comprising the steps of: 

fil (a) estimating respective off^grid brightness values at a multiplicity of off^grid positions 

^-f 2 from respective captured image brightness values at pixel positions of the pixel position array, 

■'f 3 the off-grid positions being located at respective interstices of the pixel position array and 

14 forming a regular off-grid position array; and 

15 (b) deriving pixel brightness values of an output image from the estimated off-grid 

16 brightness values. 

1 2. The method of claim 1, wherein the array of light-sensing elements of the image sensor 

2 is a linear array for capturing a line image, and the pixel position array is a linear array having 
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3 the multiplicity of pixel positions at respective regularly spaced pixel positions in a pixel row so 

4 as to define a linear captured image brightness value array. 

1 3. The method of claim 1, wherein the array of light-sensing elements of the image 

2 sensor is a linear array and the pixel position array is a two-dimensional array haivng pixel 

3 positions at respective intersections of a plurality of regularly spaced pixel rows and a plurality 

4 of regularly spaced pixel colimms , the pixel rows being orthogonal to the pixel columns, and 
q5 wherein the linear array of light-sensing elements of the image sensor is sequentially exposed to 
ln6 successive regularly spaced linear regions of the scene in multiple exposures, each one of the 
^Zj exposures producing a respective linear captured image brightness value array, the respective 
*"=^8 linear captured image brightness arrays produced by the multiple exposures being concatenated 
□9 to form a two-dimensional captured image brightness value array having respective captured 
tiO image brightness values at the pixel positions of the pixel position array. 

1 4. The method of claim 1 , wherein the image sensor has a two-dimensional array of 

2 light-sensing elements and the pixel position array is a two-dimensional array having the 

3 multiplicity of pixel positions at respective intersections of a plurality of regularly spaced pixel 

4 rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the 

5 pixel columns. 
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1 5. The method of claim 4, wherein the off-grid positions of the off-grid position array are 

2 located at respective intersections of a plurality of regularly spaced intermediate rows and a 

3 plurality of regularly spaced intermediate colunms, each one of the intermediate rows being 

4 parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows, 

5 each one of the intermediate colunms being parallel to the pixel columns and extending between 

6 a respective adjacent pair of the pixel columns. 

od 6. The method of claim 4, wherein the off-grid positions of the off-grid position array are 

^:rr2 located at respective intersections of a plurality of regularly spaced intermediate rows and the 

^"^3 pixel columns, and at respective intersections of a plurality of regularly spaced intermediate 

'^j^ columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows 

j';j5 and extending between a respective adjacent pair of the pixel rows, each one of the intermediate 

i'y6 columns being parallel to the pixel columns and extending between a respective adjacent pair of 

'TtJ the pixel columns. 

1 7. The method of claim 3, wherein the off-grid positions of the off-grid position array are 

2 located at respective intersections of a plurality of regularly spaced intermediate columns and the 

3 pixel rows, each one of the intermediate columns being parallel to the pixel columns and 

4 extending between a respective adjacent pair of the pixel columns. 
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1 8. The method of claim 2, wherein the off-grid positions of the off-grid position array are 

2 located between respective adjacent pairs of pixel positions of the linear pixel position array. 

1 9. The method of claim 1, wherein the step of deriving the output image from the 

2 estimated off-grid brightness values comprises using the estimated off-grid brightness values 

3 directly as the pixel brightness values of the output image. 

i"il 10. The method of claim 1 , wherein the step of deriving pixel brightness values of the 

^■fU output image from the estimated off-grid brightness values comprises resampling the estimated 

%3 off-grid brightness values to derive respective resampled on-grid brightness values at the pixel 

;ji4 positions of the pixel position array, the respective resampled on-grid brightness values at the 

C.35 pixel positions of the pixel position array being the pixel brightness values of the output image. 

^ 1 1 . The method of claim 5, wherein the predetermined sensitivity pattern of the array of 

2 light-sensing elements has four different predetermined sensitivity levels and corresponds to a 

3 pixel position array having repetitive groups of four nearest neighbor pixel positions, each one of 

4 the four pixel positions in each group corresponding to a different one of the four predetermined 

5 sensitivity levels in a conmion predetermining positional order. 
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1 12. The method of claim 6, wherein the predetermined sensitivity pattern of the array of 

2 light-sensing elements has first and second predetermined sensitivity levels^ and wherein 

3 adjacent pixel positions in each one of the pixel rows and adjacent pixel positions in each one of 

4 the pixel columns correspond to different ones of the first and second predetermined sensitivity 

5 levels. 

1 13. The method of claim 1, wherein the predetermined sensitivity pattern of the array of 

fi2 light-sensing elements have first and second predetermined sensitivity levels, and adjacent pixel 

Lfl3 positions in each one of the pixel rows correspond to different ones of the predetermined first and 

' f 4 second sensitivity levels, and adjacent pixel positions in each one of the pixel columns 

^'53 correspond to the same one of the predetermined first and second sensitivity levels. 

rUl 14. The method of claim 1, wherein the step of estimating respective off-grid brightness 

values at the off-grid positions of the off-grid position array from respective captured image 

3 brightness values at the pixel positions of the pixel position array comprises the steps of: (1) 

4 estimating a sensitivity level off-grid brightness value for each different one of the sensitivity 

5 levels at each one of the off-grid positions using only captured image brightness values at pixel 

6 positions corresponding to the different one of the sensitivity levels; and (2) combining the 

7 sensitivity level off-grid brightness value estimated for each different one of the sensitivity levels 
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8 at each one of the off-grid positions to derive a respective estimated off-grid brightness values at 

9 each one of the off-grid positions. 

1 15. The method of claim 1 4, v^herein the step of combining the sensitivity level of 

2 brightness value for each different one of the sensitivity levels at each one of the off-grid 

3 positions comprises adding the sensitivity level off-grid brightness values estimated for each 

4 different one of the sensitivity levels at each one of the off-grid positions to derive the respective 
estimated off-grid brightness value at each one of the off-grid positions. 

16. The method of claim 15, wherein the step of deriving pixel brightness values of the 
output image from the estimated off-grid brightness values comprises interpolating the estimated 
f ~3 off-grid brightness values to derive respective interpolated on-grid brightness values at the pixel 
11:4 positions of the pixel position array, and compensating each one of the interpolated on-grid 
;:H5 brightness values by the inverse of a combined response function of the light-sensing elements to 

6 obtain the pixel brightness values of the output image, the combined response function being the 

7 sum of weighted response functions of light-sensing elements having different ones of the 

8 sensitivity levels, each one of the weighted response functions being a response function of a 

9 light-sensing element having a respective one of the sensitivity levels multiplied by a 
10 predetermined weighting factor for the respective one of the sensitivity levels. 
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1 17. The method of claim 16, wherein the step of compensating each one of the 

2 interpolated on-grid brightness values by the inverse of the combined response function of the 

3 light-sensing elements comprises deriving from each one of the interpolated on-grid brightness 

4 values a respective index for a lookup table memory storing lookup table data representing the 

5 inverse of the combined response function of the light-sensing elements^ retrieving from the 

6 lookup table memory a compensated interpolated on-grid brightness value corresponding to each 

7 index, and providing the compensated interpolated on-grid brightness values as the pixel 
rgS brightness values of the output image. 

^'^1 18. The method of claim 14, wherein the step of combining the sensitivity level off-grid 

, r|2 brightness value for each different one of the sensitivity levels at each one of the off-grid 

i;33 positions comprises the steps of: (1) multiplying the sensitivity level off-grid brightness value for 

i y4 each different one of the sensitivity levels at each one of the off-grid positions with a respective 

;:i5 predetermined weighting factor corresponding to the different one of the sensitivity levels to 

6 derive a weighted sensitivity level off-grid brightness value for each different one of the 

7 sensitivity levels at each one of the off-grid positions; and (2) adding the weighted sensitivity 

8 off-grid level brightness value for each different one of the sensitivity levels at each one of the 

9 off-grid positions to derive the respective estimated off-grid brightness value at each one of the 
1 0 off-grid positions , 
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1 1 9. The method of claim 1 8, wherein the step of deriving the pixel brightness values of 

2 the output image from the estimated off-grid brightness values comprises interpolating the 

3 estimated off-grid brightness values to derive respective interpolated on-grid brightness values at 

4 the pixel positions of the pixel position array, and compensating each one of the interpolated on- 

5 grid brightness values by the inverse of a combined response function of the light-sensing 

6 elements to derive the pixel brightness values of the output image, the combined response 

7 function being the sum of weighted response functions of the light-sensing elements having 
different ones of the sensitivity levels, each one of the weighted response functions being a 
response function of a light-sensing element having a respective one of the sensitivity levels 

•TO multiplied by a predetermined weighting factor for the respective one of the sensitivity levels. 

[31 20. The method of claim 19 wherein the step of compensating each one of the 

: ^ interpolated brightness values by the inverse of a combined response function of the light- 

:'^3 sensing elements comprises using each one of the interpolated on-grid brightness values to derive 

4 a respective index for a lookup table memory storing data representing the inverse of the 

5 combined response function of the light-sensing elements, retrieving from the lookup table 

6 memory a respective compensated interpolated on-grid brightness value corresponding to each 

7 index, and providing the compensated interpolated on-grid brightness values as the pixel 

8 brightness values of the output image. 
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1 21 . The method of claim 1 , wherein the step of estimating respective off-grid brightness 

2 values at the off-grid positions of the off-grid position array from respective captiired image 

3 brightness values at the pixel positions of the pixel position array comprises the steps of: (1) 

4 estimating a sensitivity level off-grid brightness value for each different one of the sensitivity 

5 levels at each one of the off-grid positions using only the captured image brightness values at 

6 pixel positions corresponding to the different one of the sensitivity levels; (2) combining the 

7 sensitivity level off-grid brightness value estimated for each different one of the sensitivity levels 
qS at each one of the off-grid positions to derive a combined sensitivity level off-grid brightness 
i,ri9 value at each one of the off-grid positions; and (3) compensating the combined sensitivity level 
^'f 0 off-grid brightness value at each one of the off-grid positions by the inverse of a combined 

"3^1 response function of the light-sensing elements to derive a respective estimated off-grid 

|2 brightness value at each one of the off-grid positions, the combined response function being the 

ri3 sum of weighted response functions of light- sensing elements having different ones of the 

;=| 4 sensitivity levels, each one of the weighted response functions being a response function of a 

15 light-sensing element having a respective one of the sensitivity levels multiplied by a 

16 predetermined weighting factor for the respective one of the sensitivity levels. 

1 22. The method of claim 21 , wherein the step of compensating the combined sensitivity 

2 level off-grid brightness value at each one of the off-grid positions by the inverse of the 

3 combined response function of the light-sensing elements comprises using the combined 
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4 sensitivity level off-grid brightness value at each one of the off-grid positions to derive a 

5 respective index for a lookup table memory storing data representing the inverse of the combined 

6 response function of the light-sensing elements, retrieving from the lookup table a compensated 

7 combined sensitivity level off-grid brightness value corresponding to each index, and providing 

8 the compensated combined sensitivity level off-grid brightness values retrieved from the lookup 

9 table as the respective estimated off-grid brightness values at the off-grid positions. 

fjl 23. The method of claim 21, wherein the step of combining the sensitivity level off-grid 

\JU brightness value estimated for each different one of the sensitivity levels at each one of the off- 

^'"3 grid positions comprises adding the sensitivity level off-grid brightness value for each different 

^'p^4 one of the sensitivity levels at each one of the off-grid positions to derive the respective 

o5 combined sensitivity level off-grid brightness values at the off-grid positions. 

24. The method of claim 21, wherein the step of combining the sensitivity level off-grid 

2 brightness value estimated for each different one of the sensitivity levels at each one of the off- 

3 grid positions comprises the steps of: (1) multiplying the sensitivity level off-grid brightness 

4 value estimated for each different one of the sensitivity levels at each one of the off-grid 

5 positions by a respective predetermined weighting factor corresponding to the different one of 

6 the sensitivity levels to derive a weighted sensitivity level off-grid brightness value for each 

7 different one of the sensitivity levels at each one of the off-grid positions; and (2) summing the 
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8 weighted sensitivity off-grid level brightness value for each different one of the sensitivity levels 

9 at each one of the off-grid positions to derive the combined sensitivity level off-grid brightness 
10 value at each one of the off-grid positions. 

1 25. The method of claim 14 or 21, wherein the step of estimating the sensitivity level off- 

2 grid brightness value for each different one of the sensitivity levels at each one of the off-grid 

3 positions comprises the steps of: (1) masking the pixel positions but leaving unmasked pixel 
positions corresponding to the different one of the sensitivity levels; and (2) estimating the 

iJl5 sensitivity level off-grid brightness value for the different one of the sensitivity levels at the one 

=•'^^6 of the off-grid positions to be equal to the captured image brightness value at the one of the 

''p7 unmasked pixel positions nearest the one of the off-grid positions. 

rUl 26. The method of claim 24, wherein the array of light-sensing elements of the image 

^:^2 sensor is a linear array and the pixel position array is a linear array having pixel positions at 

3 respective regularly spaced pixel positions in a pixel row and the off-grid positions of the off- 

4 grid position array are each located between respective pairs of adjacent pixel positions of the 

5 linear pixel position array, and wherein the step of estimating a sensitivity level off-grid 

6 brightness value for each different one of the sensitivity levels at each one of the off-grid 

7 positions comprises providing as the sensitivity level off-grid brightness value for each different 

8 one of the sensitivity levels at each one of the off-grid positions the captured image brightness 
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9 values at pixel positions corresponding to the different one of the sensitivity levels that is the 

10 nearest neighbor to the one of the off-grid positions. 

1 27. The method of claim 26, wherein the predetermined sensitivity pattern of the light- 

2 sensing elements has first and second predetermined sensitivity levels, and adjacent pixel 

3 positions of the liner pixel position array correspond to different ones of the first and second 

4 predetermined sensitivity levels, and wherein the step of multiplying the sensitivity level off-grid 
rnS brightness value for each different one of the sensitivity levels at each one of the off-grid 

yi6 positions by a respective predetermined weighting factor corresponding to the different one of 

=""7 the sensitivity levels to derive a weighted sensitivity level off-grid brightness value for each 

'^j^ different one of the sensitivity levels at each one of the off-grid positions, and the step of 

j;:i9 summing the weighted sensitivity level off-grid brightness value for each different one of the 

I iO sensitivity levels at each one of the off-grid positions to derive the combined sensitivity level off- 

:::n grid brightness value at each one of the off-grid positions is expressed as 

12 B(x') = FFj/(x' " 0.5) + fr//x' + 0.5), 

13 where B(x') is the combined sensitivity level off-grid brightness value at off-grid position x', 

14 is the predetermined weighting factor for the sensitivity level corresponding to pixel position 

15 (x' - 0.5), W2 is the weighting factor for the sensitivity level corresponding to pixel position 

16 (x' + 0.5), Ic(x' - 0.5) is the captured image brightness value at pixel position (x' - 0.5) and 
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17 Ic(x' + 0.5) is the captured image brightness value at pixel position (x' + 0.5), and wherein the 

1 8 combined response function of the light-sensing elements is expressed as 

19 S(E) - W/^iE) + W/^(E), 

20 where Pi(E) is the radiometric response function of a light-sensing element having the sensitivity 

21 level corresponding to pixel position (x' - 0.5) and P2(E) is the radiometric response function of 

22 a light-sensing element having the sensitivity level corresponding to pixel position (x' + 0.5). 

Qil 28. The method of claim 1, wherein the step of estimating respective off-grid brightness 

i J=2 values at the off-grid positions of the off-grid position array from respective captured image 

^'™3 brightness values at the pixel positions of the pixel position array comprises bi-linear sampling 

"r^4 with interpolation of the captured image brightness values at the pixel positions. 

F Ul 29. The method of claim 1, wherein the step of estimating respective off-grid brightness 

i;f2 values at the off-grid positions of the off-grid position array from respective captured image 

3 brightness values at the pixel positions of the pixel position array comprises bi-cubic sampling 

4 with interpolation of the captured image brightness values at the pixel positions. 

1 30. The method of claim 1, wherein the step of estimating respective off-grid brightness 

2 values at the off-grid positions of the off-grid position array from respective captured image 

3 brightness values at the pixel positions of the pixel position array comprises estimating a a 
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4 respective off-grid brightness value at each individual one of the off-grid positions by comparing 

5 a respective captured image brightness value at each one of a predetermined number of pixel 

6 positions that are nearest neighbors to the individual one of the off-grid positions with a 

7 predetermined low threshold value and a predetermined high threshold value, computing the sum 

8 of the nearest neighbor pixel position captured image brightness values that are greater than the 

9 predetermined low threshold value and less than the predetermined high threshold value, and 

1 0 dividing the sum by the number of nearest neighbor pixel position brightness values included in 

r|l the sum. 

^Zl 31. The method of claim 1, wherein the step of estimating off-grid brightness values at 

'^j^ the off-grid positions of the off-grid position array from respective captured image brightness 

l33 values at the pixel positions of the pixel position array comprises estimating a respective off-grid 

i 04 brightness value at each individual one of the off-grid positions by comparing a respective 

;L: 5 compensated captured image brightness value at each one of a predetermined number of pixel 

6 positions that are nearest neighbors to the individual one of the off-grid positions with a 

7 predetermined low threshold value a predetermined high threshold value, computing the sum of 

8 the compensated captured image brightness values at the nearest neighbor pixel positions that are 

9 greater than the predetermined low threshold value and less than the predetermined high 

10 threshold value, and deriving a respective estimated off-grid brightness value at the individual 

1 1 one of the off-grid positions by dividing the sum by the number of compensated captured image 
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12 brightness values at nearest neighbor pixel positions, the respective compensated captured image 

13 brightness value at each one of the nearest neighbor pixel positions being the captured image 

14 brightness value at the one of the nearest neighbor pixel positions compensated by the inverse of 

15 the response function of a light-sensing element having a sensitivity level corresponding to the 

16 one of the nearest neighbor pixel positions. 

1 32. The method of claim 3 1 , wherein the respective compensated captured image 

Q=2 brightness value at each one of the nearest neighbor pixel positions is derived by using the 

=::n3 captured image brightness value at the one of the nearest neighbor pixel positions to derive an 

•"14 index for a lookup table memory storing data representing the inverse response functions of 

;}i5 light-sensing elements having different ones of the plurality of sensitivity levels, and retrieving 

1:36 from the lookup table memory a compensated captured image brightness value corresponding to 

i '-^7 the index from lookup table data representing the inverse of the response function of a light- 

;::f 8 sensing element having the sensitivity level of the one of the nearest neighbor pixel positions. 

1 33. The method of claim 5, wherein each one of the intermediate rows extends medially 

2 between a respective adjacent pair of the pixel rows and each one of the intermediate columns 

3 extends medially between a respective adjacent pair of the pixel colunms, and wherein the step of 

4 estimating respective off-grid brightness values at the off-grid positions of the off-grid position 

5 array from respective captured image brightness values at the pixel positions of the pixel position 
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6 array comprises estimating the brightness value at each one of the off-grid positions (x',y'), 

7 where has values from 1 to xSize- 1, y' has values from 1 to ySize- 1, and xSize and ySize are 

8 the dimensions of the captured image in the direction of the pixel rows and the pixel columns, 

9 respectively, the step of estimating respective off-grid brightness values at each one of the off- 

10 grid positions (x',y') comprising: 

1 1 (i) comparing a respective brightness value I(u,v) at each one of sixteen pixel 

12 positions (u,v) that are nearest neighbors to the one of the off-grid positions (x\y') with a 

i:|3 predetermined low threshold value and a predetermined high threshold value, where u has values 

from x' - 5 to x' + 0.5 and v has values from y ' - 0.5 to y' + 0.5; 
^'£5 (ii) for each one of the brightness values I(u,v) at the sixteen nearest neighbor 

. pixel positions (u,v) that is greater than the predetermined low threshold value and less than the 

£17 predetermined high threshold value, computing indices i and j of a 4 x 4 cubic interpolation 

^ ys kernel (|) using the relations i = x'-u- 1.5 and j = y'-v- 1.5, and computing the quantities 

a ^2 1 ^"'^ 



ab 



19 and (|)^, where <j)^^ is the value of the 4 x 4 cubic interpolation kernel ({) at indices m and n; 

20 (iii) adding the quantities 



2 ^ji 



ab 
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21 computed in step (ii) to derive a first sum s; 

22 (iv) adding the quantities cj)^. computed in step (ii) to derive a second sum w; and 

23 (v) computing the estimated brightness value at the one of the off-grid positions 

24 (x^y') by dividing the first sum s by the second sum w. 

1 34. The method of claim 33^ wherein the brightness value ^u^v) at nearest neighbor pixel 

2 position (u^v) compared with the predetermined low threshold and the predetermined high 
threshold value in step (i) is the captured image brightness value at pixel position (u,v). 

i"^l 35. The method of claim 33, wherein the brightness value I(u,v) at nearest neighbor pixel 

' ;L2 position (u,v) compared with the predetermined low threshold value and the predetermined high 

r-3 threshold value in step (i) is the captured image brightness value at pixel position (u,v) 

rU4 compensated by the inverse of a response function of a light-sensing element having the 

=;f 5 sensitivity level corresponding to pixel position (u,v). 

1 36. The method of claim 35, wherein the captured image brightness value at each one of 

2 the nearest neighbor pixel positions (u,v) is compensated by the inverse of the response function 

3 of a light-sensing element having the sensitivity level corresponding to one of the nearest 

4 neighbor pixel positions (u,v) by using the captured image brightness value to derive an index for 

5 a lookup table memory storing data representing the inverse of response functions of light- 
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6 sensing elements having different ones of the plurahty of sensitivity levels, and retrieving from 

7 the lookup table memory a compensated captured image brightness value corresponding to the 

8 index using lookup table data representing the inverse of the response function of a light-sensing 

9 element having the sensitivity level corresponding to the one of the nearest neighbor pixel 

10 positions (u,v), the retrieved compensated brightness value being the brightness value I(u,v) at 

1 1 the one of the nearest neighbor pixel positions (u,v). 

f-tl 37. The method of claim 2, wherein the step of estimating respective off-grid brightness 

i,n2 values at the off-grid positions of the off-grid position array from the respective captured image 

f 3 brightness values at the pixel positions of the pixel position array comprises estimating the 

^'pJ4 brightness value at each one of the off-grid positions x', where x' has values from 2 to xSize - 2, 

i:;i5 xSize being the dimension of the captured line image, the step of estimating the off-grid 

rU6 brightness value at each one of the ofp-grid positions x' comprising: 

(i) comparing a respective brightness value I(k) at each one of four pixel positions 

8 k that are nearest neighbors to the one of the off-grid positions x' with a predetermined low 

9 threshold value and a predetermined high threshold value, where k has values from x'- 1 .5 to 

10 x'+1.5; 

1 1 (ii) for each one of the brightness values I(k) at the four nearest neighbor pixel 

12 positions k that is greater than the predetermined low threshold value and less than the 
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13 predetermined high threshold value computing the quantity I(k) G(k-~x')5 where G(k-x') is the 

14 value of a 4 x 1 Gaussian interpolation kernel G at position (k-x'); 

15 (iii) adding the quantities I(k) G(k-x') computed in step (ii) to derive a first 

16 sump; 

17 (iv) adding the quantities G(k~-x') used in the computations of step (ii) to derive a 

1 8 second sum q; and 

19 (v) computing the estimated brightness value at the one of the off-grid position x' 
r£:0 by dividing the first sxmi p by the second sum q. 

^"^1 38. The method of claim 37, wherein the respective brightness value I(k) at nearest 

|j~2 neighbor pixel position k compared with the predetermined low threshold value and the 

i::;i3 predetermined high threshold value in step (i) is the captured image brightness value at nearest 

rU4 neighbor pixel position k. 

1 39. The method of claim 37, wherein the brightness value I(k) at nearest neighbor pixel 

2 position k compared with the predetermined low threshold value and the predetermined high 

3 threshold value in step (i) is the captured image brightness value at nearest neighbor pixel 

4 position k compensated by the inverse of a light-sensing element having the sensitivity level 

5 corresponding to pixel position k. 
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1 40. The method of claim 39, wherein the captured image brightness value at each one of 

2 the nearest neighbor pixel positions k is compensated by the inverse of the response function of a 

3 light-sensing element having the sensitivity level corresponding to the one of the nearest 

4 neighbor pixel positions k by using the captured image brightness value at the one of the nearest 

5 neighbor pixel positions k to an index for a lookup table memory storing data representing the 

6 inverse of response functions of light-sensing elements having different ones of the plurality of 

7 sensitivity levels, and retrieving from the lookup table memory a compensated captured image 
r:38 brightness value corresponding to the index using lookup table data representing the inverse of 
-n9 the response function of a light-sensing element having the sensitivity level corresponding to the 
^10 one of the nearest neighbor pixel positions k, the retrieved compensated captured image 

;jl|l brightness value being the brightness value I(k) at the one of the nearest neighbor pixel 

[12 position k. 

I'^l 41. The method of claim 37, wherein the 4 x 1 Gaussian kernel G has the form: 



1.0 


7.7 


7.7 


1.0 



3 

1 42. The method of claim 1 0, wherein the step of resampling the estimated off-grid 

2 brightness values to derive respective resampled on-grid brightness values at the pixel positions 
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3 of the pixel position array comprises deriving the resampled brightness value at each individual 

4 one of the pixel positions by computing the product of respective off-grid brightness values at a 

5 predetermined number of off-grid positions that are nearest neighbors to the individual one of the 

6 pixel positions and cubic interpolation kernel having the same dimensions as the predetermined 

7 number of off-grid positions. 

1 43. The method of claim 5, wherein the step of deriving pixel brightness values of an 

[32 output image from the estimated off-grid brightness values comprises resampling the estimated 

yT3 off-grid brightness values to derive resampled on-grid brightness values at the pixel positions of 

^'lA the pixel position array, and wherein each one of the intermediate rows extends medially between 

Jr3 a respective adjacent pair of pixel rows and each one of the intermediate columns extends 

r36 medially between a respective adjacent pair of pixel columns, and wherein the resampled 

ry7 brightness value at each one of the multiplicity of pixel positions is computed by the relation: 

/(x-2j;-2) = E E B(x^L5 ^ U y~L5 + ;)(|)., 

1=0 J=0 ^ 

8 where x has values from 2.5 to xSize-2.5, y has values from 2.5 to ySize -2.5, 1(x-2, y-2) is the 

9 resampled on-grid brightness value at pixel position (x--2, y-2), B(x- 1 .5+i, y- 1 .5+j) is the 

10 estimated off-grid brightness value at off-grid position (x- 1.5+i, y- 1 .5+j), i and j are indices of a 

11 4x4 cubic interpolation kernel (|); the indices i and j each have values from 0 to 3, is the 
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value of the interpolation kernel ({> at indices i and j, xSize is the dimension of the captured image 
in the direction of the pixel rows, and ySize is the dimension of the captured image in the 
direction of the pixel columns. 

44. The method of claim 33 or 43, wherein the 4 x 4 cubic interpolation kernel cj) has the 

form: 



0.043 
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0.043 


-0.661 


10.28 


10.28 


-0.661 
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45. The method of claim 2, wherein the step of deriving pixel brightness values of an 
output image from the estimated off-grid brightness values comprises resampling the estimated 
off-grid brightness values to derive resampled on-grid brightness values at pixel positions of the 
pixel position array, and wherein each off-grid position is located midway between a respective 
adjacent pair of pixel positions, and the resampled on-grid brightness values at each one of the 
pixel positions x is computed by the relation: 

k = -1.5 

where x is the coordinate of the pixel position having values from 3.5 to xSize-3.5, Io(x-3) is the 
resampled on-grid brightness value at pixel position (x-3), B(x+k) is the estimated off-grid 
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brightness value at off-grid position (x+k), k is a position of a 4 x 1 cubic interpolation kernel x[r, 
x(f (k) is the value of the cubic interpolation kernel x|r at position k, the position k has values from 
" 1 .5 to 1 .5, and xSize is the dimension of the captured line image. 

46. The method of claim 45 , wherein the 4 x 1 cubic interpolation kernel x|; has the 

form: 



-0.96 


13.44 


13.44 


-0.96 



k 



47. The method of claim 6, wherein each one of the intermediate rows extends medially 
between a respective adjacent pair of the pixel rows and each one of the intermediate columns 
extends medially between a respective adjacent pair of the pixel columns, and the step of 
estimating respective off-grid brightness values at the off-grid positions of the off-grid position 
array from respective captured image brightness values at the pixel positions of the pixel position 
array comprises estimating the off-grid brightness value at each one of the off-grid positions 
(x',y'), where x' has values from 0.5 to xSize-0.5 and y' has values from 1 to ySize- 1 for off- 
grid positions located at respective intersections of the intermediate rows and the pixel columns, 
x' has values from 1 to xSize- 1 and y' has values from 0.5 to ySize-0.5 for off-grid positions 
located at respective intersections of the pixel rows and the intermediate columns, and xSize and 
ySize are the dimensions of the captured image in the directions of the pixel rows and the pixel 
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12 columns, respectively, the step of estimating the brightness values at each one of the off-grid 

13 positions (x',y')comprising: 

14 (i) for each one of the off-grid positions (x, y') computing coordinates (u, v) using 

15 the relations u = x + 0.5i - 0.5j and v = y' - 1.5 + 0.5i +0.5j, where i and j are indices of a 4 x 4 

16 cubic interpolation kemel (j), and each one of the indices i and j has values from 0 to 3; 

17 (ii) for each pair of coordinates (u, v) computed in step (i) using indices i and j, 

1 8 determining whether the coordinates (u,v) are those of a pixel position; 

[|9 (iii) if the coordinates (u, v) computed in step (i) are the coordinates of a pixel 

^ Jo position, comparing a brightness value I(u,v) at the pixel position (u,v) with a predetermined 
low-threshold value and a predetermined high threshold value; 

(iv) if the coordinates (u, v) computed in step (i) using indices i and j are those of 

i23 a pixel position and the brightness value I(u,v) is greater than the predetermined low threshold 

ri4 value and less than the predetermined high threshold value, then compute the quantities: 

□ ^ (j).. I(u,v) 

? 

ab 

25 and where 4)m,n is the value of a 4 x 4 cubic interpolation kemel (j) at indices m and n; 

26 (v) adding the quantities 

, (|) I{u,v) 

ab 
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27 computed in step (iv) to derive a first sum s; 

28 (vi) adding the quantities <j>j/ computed in step (iv) to derive a second sum w; and 

29 (vii) computing the estimated brightness value at the one of the off-grid positions 

30 (x', y') by dividing the first sum s by the second sum v^. 

1 48. The method of claim 47, wherein the brightness value I(u,v) at each one of the pixel 

2 positions (u,v) compared with the predetermined low threshold value and the predetermined high 
Hi threshold value in step (iii) is the captured image brightness value at the one of the pixel 

W positions (u,v). 

:^r|l 49. The method of claim 47, wherein the brightness value I(u,v) at each one of the pixel 

i:32 positions (u,v) compared with the predetermined low threshold value and the predetermined high 

i ^-;3 threshold value in step (iii) is the captured image brightness value at the one of the pixel 

;;S4 positions (u,v) compensated by the inverse of the response function of a light-sensing element 

5 having a sensitivity level corresponding to the one of the pixel positions (u,v). 

1 50. The method of claim 49, wherein the captured image brightness value at each one of 

2 the pixel positions (u,v) is compensated by the inverse of the response function of a light-sensing 

3 element having the sensitivity level corresponding to the one of the pixel positions (u,v) by using 

4 the captured image brightness value Ic(u,v) at the one of the pixel positions (u,v) to derive an 
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5 index for a lookup table memory storing data representing the inverse of response functions of 

6 light-sensing elements having different ones of the plurality of sensitivity levels, and retrieving 

7 from the lookup table a compensated captured image brightness value corresponding to the index 

8 using lookup table data representing the inverse of the response function of a light-sensing 

9 element having the sensitivity level corresponding to the one of the pixel positions (u,v), the 

10 retrieved compensated captured image brightness value being the brightness value I(u,v) at the 

1 1 one of the pixel positions (u,v). 

tni 5 1 . The method of claim 47, wherein the step of estimating the off-grid brightness value 

e^ch one of the off-grid positions (x', y') is first carried out for off-grid positions located at 

p3 respective intersections of intermediate rows and pixel columns, and then carried out for pixel 

f;34 positions located at respective intersections of pixel rows and intermediate columns. 

::.;fl 52. The system of claim 47, wherein the 4 X 4 cubic interpolation kemel (j) has the form: 
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ijl 53. The method of claim 6, wherein the step of deriving pixel brightness values of an 

^ y2 output image from the estimated off-grid brightness values comprises resampling the estimated 

::::3 off-grid brightness values to derive respective resampled on-grid brightness values at pixel 

4 positions of the pixel position array, and each one of the intermediate rows extends medially 

5 between a respective adjacent pair of the pixel rows and each one of the intermediate columns 

6 extends medially between a respective adjacent pair of the pixel columns, and wherein the 

7 resampled on-grid brightness value at each one of the pixel positions (x,y) is computed by the 

8 relation: 
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3 3 

/(x-2^-2) - J2 E 5(x+0.5/-0.57>-L5+0.5/+0.5/)(l)., 

i=o j=0 ^ 

9 where x has values from 2.5 to xSize -2.5, y has values from 2.5 to ySize -2.5, 1(x-2, y-2) is 

10 the resampled on-grid brightness value at pixel position (x-2, y-2), 

1 1 B(x+0.5i- 0.5j,y- 1 .5+0.5i+0.5j) is the off-grid brightness value at off-grid position 
(x+0.5i-j0.5j, y- 1 .5+0.5i+0.5jX i and j are indices of a 4 x 4 cubic interpolation kernel (|), 

liB indices i and j each have values from 0 to 3, (})y is the value of the interpolation kernel (J) at 
='f4 indices i and j, xSize is the dimension of the captured image in the direction of the pixel rows, 
^'1^5 and ySize is the dimension of the captured image in the direction of the pixel columns. 

rUl 54. The method of claim 5, wherein the step of deriving pixel brightness values of an 

output image from the estimated off-grid brightness values comprises using a Gaussian kernel for 

3 interpolating the estimated off-grid brightness values to derive respective interpolated brightness 

4 values at the pixel positions of the pixel position array. 
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55. The method of claim 54, wherein the Gaussian kernel has the form: 
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56. The method of claim 5, wherein the step of deriving pixel brightness values of an 
output image from the estimated off-grid brightness values comprises using a bi-linear 
interpolation filter kernel for interpolation of the off-grid brightness values to derive respective 
interpolated on-grid brightness values at the pixel positions of the pixel position array. 

57. The method of claim 56, wherein the bi-linear interpolation filter kernel has the form: 
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58. The method of claim 5 or 7, wherein the step of deriving pixel brightness value of 
an output image from the estimated off-grid brightness values comprises resampling the 
estimated off-grid brightness values to derive respective resampled on-grid brightness values at 
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4 the pixel positions of the pixel position array, and wherein the step of resampling the estimated 

5 off-grid brightness values comprises shifting the off-grid position array to coincide with the pixel 

6 position array, whereby the resampled on-grid brightness values at the pixel positions of the pixel 

7 position array are the respective estimated off-grid brightness values at the coincident off-grid 

8 positions of the shifted off-grid position array. 

1 59. A method for obtaining a relatively high dynamic range image of a scene using a 

rl2 relatively low dynamic range image sensor exposed to incident light from the scene for capturing 

l.r3 an image thereof, the image sensor having a multiplicity of light-sensing elements in an array, 

■"^4 each light-sensing element having a particular one of a plurality of sensitivity levels to incident 

^ pj5 light in accordance with a predetermined sensitivity pattern for the array of light-sensing 

elements and having a respective response function, each light-sensing element being responsive 

\ ^{7 to incident light from the scene for producing a captured image brightness value at a 

;:;t;8 corresponding one of a multiplicity of pixel positions of a pixel position array, whereby each one 

9 of the multiplicity of pixel positions corresponds to a particular one of the plurality of sensitivity 

10 levels of the light-sensing elements, the method comprising the steps of computing respective 

1 1 on-grid brightness values at pixel positions of the pixel position array from respective captured 

12 image brightness values at the pixel positions of the pixel position array, each one of the 

13 respective on-grid brightness values being computed from a corresponding plurality of the 
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14 captured image brightness values, and deriving pixel brightness values of an output image from 

1 5 the on-grid brightness values. 

1 60. The method of claim 59, wherein the array of light-sensing elements of the image 

2 sensor is a linear array for capturing a line image, and the pixel position array is a linear array 

3 having the multiplicity of pixel positions at respective regularly spaced pixel positions in a pixel 

4 row so as to form a linear captured image brightness value array. 

Ull 6L The method of claim 59, wherein the array of light- sensing elements of the image 

sensor is a linear array and the pixel position array is a two-dimensional array having pixel 

j'S positions at respective intersections of a plurality of regularly spaced pixel rows and a plurality 

i^'l4 of regularly spaced pixel colimms, the pixel rows being orthogonal to the pixel columns, and 

wherein the linear array of light-sensing elements of the image sensor is sequentially exposed to 

.L:f6 successive regularly spaced linear regions of the scene in multiple exposures, each one of the 

7 exposures producing a respective linear captured image brightness value array, the respective 

8 linear captured image brightness value arrays produced by the multiple exposures being 

9 concatenated to form a two-dimensional captured image brightness value array having respective 
10 captured image brightness values at the pixel positions of the pixel position array. 
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1 62. The method of claim 59, wherein the image sensor has a two-dimensional array of 

2 light-sensing elements, and the pixel position array is a two-dimensional array having the 

3 multiplicity of pixel positions at respective intersections of a plurality of regularly spaced pixel 

4 rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the 

5 pixel columns. 

1 63. The method of claim 59, wherein the step of deriving pixel brightness values of an 

output image from the on-grid brightness values comprises using the respective on-grid 

iJB brightness values at the pixel positions of the pixel position array as the pixel brightness values 

'^'f 4 of the output image. 

i:;ll 64. The method of claim 59, wherein the step of deriving pixel brightness values of an 

rU2 output image from the on-grid brightness values comprises compensating the respective on-grid 

^;!r:3 brightness value at each one of the pixel positions of the pixel position array by the inverse of a 

4 combined response function of the light-sensing elements to obtain respective pixel brightness 

5 values of the output image at the pixel positions of the pixel position array, the combined 

6 response function being the sum of weighted response functions of the light-sensing elements 

7 having different ones of the plxjrality of sensitivity levels, each one of the weighted response 

8 functions being the response function of a light-sensing element having a respective one of the 
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9 sensitivity levels multiplied by a predetermined weighting factor for the respective one of the 
1 0 sensitivity levels. 

1 65. The method of claim 64, wherein the step of compensating the on-grid brightness 

2 value at each one of the pixel positions of the pixel position array comprises the steps of deriving 

3 from the on-grid brightness value at each one of the pixel positions a respective index for a 

4 lookup table memory storing data representing the inverse of the combined response function of 
the light-sensing elements, and retrieving from the lookup table memory a compensated on-grid 

y16 brightness value corresponding to the respective index, the compensated on-grid brightness value 
^'17 being the pixel brightness value of the output image at the one of the pixel positions. 

1:31 66. The method of claim 59, wherein the step of computing respective on-grid brightness 

ni2 values at pixel positions of the pixel position array comprises computing the respective on-grid 
brightness value at each one of the pixel positions by calculating the product of the captured 

4 image brightness values at a predetermined number of pixel positions that are nearest neighbors 

5 to the one of the pixel positions and an on-grid interpolation filter kernel having the same 

6 dimensions as the predetermined number of pixel positions. 

1 67. The method of claim 66, wherein the image sensor has a two-dimensional array of 

2 light-sensing elements, and the pixel array is a two-dimensional array having the multiplicity of 
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3 pixel positions at respective intersections of a plurality of regularly spaced pixel rows and a 

4 plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel 

5 columns, and wherein the on-grid brightness value at each one of the pixel positions (x,y) of the 

6 pixel position array is calculated by the relation 

I{x-2,y-2) = E E /(x-2 ^ Uy-2 ^ 

i=o 7=0 ^ 

i;;;37 whcrc X has values from 2.5 to xSize - 2.5, y has values from 2.5 to ySize - 2.5, 1(x - 2, y - 2) is 

ln8 the interpolated brightness value at pixel position (x - 2, y - 2), I^Cx - 2 + i, y - 2 + j) is the 

■''^9 captured image brightness value at pixel position (x - 2 + i, y - 2 + j), i and j are indices of a 5 x 5 

' on-grid interpolation fdter kernel (j), the indices i and j each have values from 0 to 4, (j)^ is the 

i:|l value of the on-grid interpolation filter kemel (j) at indices i and j, xSize is the dimension of the 

r3l2 captured image in the direction of the pixel rows, and ySize is the dimension of the captured 

::|3 image in the direction of the pixel columns. 
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1 68. The method of claim 67, wherein the 5 x 5 on-grid interpolation filter kernel c|) has 

2 the form: 
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=;K1 69. The method of claim 59, wherein the step of computing respective on-grid brightness 

2 values at pixel positions of the pixel position array comprises computing a respective on-grid 

i yS brightness value at each individual one of the pixel positions of the pixel position array including 

|;J^4 the steps of : 

"5 (i) comparing a respective compensated captured image brightness value at each one of a 

6 predetermined number of pixel positions that are nearest neighbors to the individual one of the 

7 pixel positions with a predetermined low threshold value and a predetermined high threshold 

8 value; 

9 (ii) for each one of the nearest neighbor pixel positions having a compensated captured 

1 0 image brightness value greater than the predetermined low threshold value and less than the 

1 1 predetermined high threshold value, computing the product of the compensated captured image 
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12 brightness value at the one of the nearest neighbor pixel positions and the value of an on-grid 

13 interpolation filter kernel corresponding to the one of the nearest neighbor pixel positions, the 

14 on-grid interpolation filter kernel having the same dimensions as the predetermined number of 

1 5 nearest neighbor pixel positions; 

16 (iii) computing a first sum of all products computed in step (ii); 

17 (iv) computing a second sum of all values of the on-grid interpolation filter kernel used 

18 to compute a product in step (ii); and 

(v) computing the on-grid brightness value at the individual one of the pixel positions by 

i-M) dividing the first sum by the second sum, wherein the respective compensated captured image 

^'21 brightness value at each one of the nearest neighbor pixel positions is the respective captured 

jS2 image brightness value at the one of the nearest neighbor pixel positions compensated by the 

j:33 inverse of the response fianction of a light-sensing element having the sensitivity level 

ri4 corresponding to the one of the nearest neighbor pixel positions. 

1 70. The method of claim 69, further comprising the steps of using the captured image 

2 brightness value at each one of the pixel positions of the pixel position array to derive a 

3 respective index for a lookup table storing data representing the inverse of response functions of 

4 light-sensing elements having different ones of the plurality of sensitivity levels, and retrieving 

5 from the lookup table a compensated captured image brightness value corresponding to the 

6 respective index fi-om lookup table data representing the inverse of the response function of the 
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7 light-sensing element having the sensitivity level corresponding to the one of the pixel positions, 

8 the retrieved compensated captured image brightness value being the compensated captured 

9 image brightness value at the one of the pixel positions. 

1 71 . A method for obtaining a relatively high dynamic range image of a scene using a 

2 relatively low dynamic range image sensor exposed to incident light from the scene for capturing 

3 an image thereof, the image sensor having a multiplicity of light-sensing elements in a linear 
array, each light-sensing element having a particular one of a plurality of sensitivity levels to 

Lr6 incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing 

^"^6 elements and having a respective response function, each light-sensing element being responsive 

"rf%] to incident light from the scene for producing a captured image brightness value at a 

CiS corresponding one of a multiplicity of pixel position at respective regularly spaced pixel 

n-i9 positions of a linear pixel position array to form a linear captured image brightness value array, 

0 whereby each one of the multiplicity of pixel positions corresponds to a particular one of the 

1 1 plurality of sensitivity levels, the method comprising the steps of: 

12 (i) for each one of the pixel positions x of the pixel position array, where x has 

13 values from 2.5 to xSize-2.5 and xSize is the dimension of the captured image, comparing each 

14 one of brightness values I(k) at five pixel positions k that are nearest neighbors to pixel position 

15 X with a predetermined low threshold value and a predetermined high threshold value, where k 

1 6 has values from x- 2 to x+2 ; 
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17 (ii) for each one of the brightness values I(k) at the five nearest neighbor pixel 

18 positions k that is greater than the predetermined low threshold value and less than the 

19 predetermined high threshold value, computing the quantity I(k)G(k-x), vs^here G(k-x) is the 

20 value of a Gaussian interpolation kernel G at position (k-x); 

21 (iii) adding the quantities I(k)G(k-x) computed in step (ii) to derive a first sum p; 

22 (iv) adding the Gaussian interpolation kernel values G(k-x) for all values of k 

23 where I(k) is greater than the predetermined low threshold value and less than the predetermined 
i5|4 high threshold value to derive a second sum q; and 

^ J5 (v) computing a pixel brightness value of an output image I^Cx) at pixel position x 

by dividing the first sum p by the second sum q. 

j;3l 72. The method of claim 71, wherein the brightness value I(k) at pixel position k 

rU2 compared with the predetermined low threshold value and the predetermined high threshold 
value in step (i) is the captured image brightness value at pixel position k. 

1 73. The method of claim 71, wherein the brightness value I(k) at pixel position k 

2 compared with the predetermined low threshold value and the predetermined high threshold 

3 value in step (i) is the captured image brightness value at pixel position k compensated by the 

4 inverse of response function of a light-sensing element having a sensitivity level corresponding 

5 to pixel position k. 
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1 74. The method of claim 73, wherein the captured image brightness value at pixel 

2 position k is compensated by the inverse of the response function of a light-sensing element 

3 having a sensitivity level corresponding to pixel position k by using the captured image 

4 brightness value at pixel position k to derive a respective index for a lookup table memory 

5 storing data representing the inverse of response functions of light-sensing elements having 

6 different ones of the plurality of sensitivity levels, and retrieving from the lookup table memory a 
r-J compensated captured image brightness value I(k) corresponding to the respective index using 
ins lookup table data representing the inverse of the response function of a light-sensing element 
•'^9 having the sensitivity level corresponding to pixel position k. 

rjl 75. The method of claim 72, wherein the 5 x l Gaussian kernel G has the form: 
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1 76. The method of claim 3 1 , 35, 39, 49, 69 or 73, wherein the predetermined low 

2 threshold value is the noise level of a light-sensing element having the sensitivity level 

3 corresponding to the pixel position having the compensated brightness value being compared 

4 therewith compensated by the inverse of the response function of a light-sensing element having 
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5 the corresponding sensitivity level, and the predetermined high threshold value is the saturation 

6 or near saturation brightness value of a light-sensing element having the same sensitivity level. 

1 77. A system for obtaining capturing a relatively high dynamic range image of a scene 

2 using a relatively low dynamic range image sensor adapted to be exposed to incident light from 

3 the scene for capturing an image thereof comprising: 

4 (a) an image sensor having a multiplicity of light-sensing elements in an array, each light- 
q5 sensing element having a particular one of a plurality of sensitivity levels to incident light in 

ln6 accordance with a predetermined sensitivity pattern for the array of light-sensing elements and a 

^""7 respective response function, each light-sensing element being responsive to incident light from 

'^fS the scene for producing a captixred image brightness value at a corresponding one of a 

multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity of 

riO pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light- 

1 sensing elements ; 

12 (b) a captured image memory for storing the captured image brightness values produced 

13 by the light-sensing elements at corresponding ones of the multiplicity of pixel positions of the 

1 4 pixel position array ; 

15 (c) an off-grid estimator for deriving from the captured image brightness values in the 

16 captured image memory respective estimated off-grid brightness values at a multiplicity of off- 
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17 grid positions located at respective interstices of the pixel position array so as to form a regular 

1 8 off-grid position array; and 

19 (d) an output image generator for deriving pixel brightness values of an output image 

20 from the estimated off-grid brightness values. 

1 78. The system of claim 77, wherein the array of light-sensing elements of the image 

2 sensor is a linear array for capturing a line image and the pixel position array is a linear array 
□3 having the multiplicity of pixel positions located at respective regularly spaced positions in a 
104 pixel row so as to form a linear captured image brightness value array, the captured image 

memory for storing the captured image brightness values produced by the light-sensing elements. 

Ql 79. The system of claim 78, wherein the off-grid positions of the off-grid position array 

are located between respective adjacent pairs of pixel positions of the linear pixel position array. 

1 80. The system of claim 79, wherein the predetermined sensitivity pattern has a 

2 predetermined first sensitivity level and a predetermined second sensitivity level, and adjacent 

3 pixel positions of the luiear pixel position array correspond to different ones of the predetermined 

4 first sensitivity level and the predetermined second sensitivity level. 
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1 81. The system of claim 77, wherein the array of light-sensing elements of the image 

2 sensor is a linear array and the pixel position array is a two-dimensional array having pixel 

3 positions at respective intersections of a plurality of regularly spaced pixel rows and a plurality 

4 of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and 

5 wherein the linear array of light-sensing elements of the image sensor is adapted to be 

6 sequentially exposed to successive regularly spaced linear regions of the scene in multiple 

7 exposures, each one of the exposures producing a respective linear captured image brightness 
value array, the respective linear captured image brightness value arrays produced by the 

\I9 multiple exposures being concatenated to form a two-dimensional captured image brightness 

^fO value array having respective captured image brightness values at the pixel positions of the pixel 

iii position array, the concatenated linear captured image brightness value arrays being stored in the 

:+2 captured image memory as respective captured image brightness values at the pixel positions of 

rlB the two-dimensional pixel position array. 



1 82. The system of claim 81, wherein the off-grid positions of the off-grid position array 

2 are located at respective intersections of a plurality of regularly spaced intermediate columns and 

3 pixel rows, each one of the intermediate columns being parallel to the pixel columns and 

4 extending between a respective adjacent pair of the pixel columns. 
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1 83. The system of claim 82, wherein the predetermined sensitivity pattern has a first 

2 predetermined sensitivity level and a second predetermined sensitivity level, and wherein 

3 adjacent pixel positions in each one of the pixel rows correspond to different ones of the first and 

4 second predetermined sensitivity levels, and adjacent pixel positions in each one of the pixel 

5 columns correspond to the same one of the first and second predetermined sensitivity levels. 

1 84. The system of claim 77, wherein the image sensor has a two-dimensional array of 
r-r2 light-sensing elements and the multiplicity of pixel positions are located at respective 

LfB intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel 

columns, the pixel rows being orthogonal to the pixel columns, the captured image memory for 

^i=^5 storing the captured image brightness values produced by the light-sensing elements at 

i7;j6 corresponding ones of the multiplicity of pixel positions of the pixel position array. 

^;:^1 85. The system of claim 84 wherein the off-grid positions of the off-grid position array 

2 are located at respective intersections of a plurality of regularly spaced intermediate rows and a 

3 plurality of regularly spaced intermediate columns, each intermediate row being parallel to the 

4 pixel rows and extending between a respective adjacent pair of pixel rows and each intermediate 

5 column being parallel to the pixel columns and extending between a respective adjacent pair of 

6 pixel columns. 



NY02.241764.1 



- 125- 



FILE NO. A32708-070050.1 125 



1 86. The system of claim 85, wherein the predetermined sensitivity pattern of the array of 

2 light-sensing elements has four different predetermined sensitivity levels and corresponds to a 

3 repetitive pattem of groups of four nearest neighbor pixel positions, each one of the four pixel 

4 positions in each group corresponding to a different one of the four predetermined sensitivity 

5 levels in a common predetermined positional order. 

1 87. The system of claim 84, wherein the off-grid positions of the off-grid position array 

rfi are located at respective intersections of a plurality of regularly spaced intermediate rows and the 

i.ilS plurality of pixel columns, and at respective intersections of a plurality of regularly spaced 

^^^^4 intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the 

;'p^5 pixel rows and extending between a respective adjacent pair of the pixel rows, each one of the 

i;:i6 intermediate columns being parallel to the pixel columns and extending between a respective 

rU7 adjacent pair of pixel columns. 

1 88, The system of claim 87, wherein the predetermined sensitivity pattem of the array of 

2 light-sensing elements has first and second predetermined sensitivity levels, and wherein 

3 adjacent pixel positions in each one of the pixel rows and adjacent pixel positions in each one of 

4 the pixel colvimns correspond to different ones of the first and second predetermined sensitivity 

5 levels. 
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1 89. The system of claim 77, wherein the output image generator comprises an output 

2 image memory for storing the respective estimated off-grid brightness values derived by the off- 

3 grid estimator as the pixel brightness values of the output image. 

1 90. The system of claim 77, wherein the output image generator comprises: 

2 (a) an off-grid brightness value memory for storing the estimated off-grid brightness 

3 values derived by the off-grid estimator; 

f 34 (b) an on-grid resampler for deriving from the estimated off-grid brightness values in the 

lfl5 off-grid brightness value memory respective resampled on-grid brightness values at the pixel 

='^6 positions of the pixel position array; and 

^1=^7 (c) an output image memory for storing the respective resampled on-grid brightness 

riS values derived by the on-grid resampler as the pixel brightness values of the output image. 

|::f 1 91 . The system of claim 77, wherein the off-grid estimator comprises: 

2 (a) an off-grid position generator for providing the off-grid positions of the off-grid 

3 position array; 

4 (b) a sensitivity pattem memory for storing data indicative of the sensitivity level 

5 corresponding to each one of the pixel positions of the pixel position array; 

6 (c) a plurality of sensitivity level off-grid brightness value estimators each corresponding 

7 to a respective one of the plurality of sensitivity levels of the light-sensing elements, and 
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8 receiving off-grid positions from the off-grid position generator and sensitivity pattern data from 

9 the sensitivity pattern memory, each one of the sensitivity level off-grid brightness value 

10 estimators being responsive to an off-grid position received from the off-grid position generator 

1 1 and the sensitivity pattern data from the sensitivity pattern memory for deriving from the 

12 captured image brightness values in the captured image memory a respective sensitivity level 

13 off-grid brightness value for the corresponding sensitivity level at the received off-grid position; 

14 and 

r|5 (d) an accumulator for combining the respective sensitivity level off-grid brightness 

lfl6 values derived by the plurality of sensitivity level off-grid brightness value estimators for the 

^'f? corresponding sensitivity levels at each off-grid position received from the off-grid position 

.'lj8 generator to derive a respective estimated off-grid brightness value at each off-grid position 

i;|9 received from the off-grid position generator. 

92. The system of claim 90, wherein the on-grid resampler comprises: 

2 (i) a pixel position generator for providing the pixel positions of the pixel 

3 position array; 

4 (ii) an interpolator for receiving pixel positions provided by the pixel position 

5 generator and being responsive to a pixel position received from the pixel position generator for 

6 deriving from the estimated off-grid brightness values in the off-grid brightness value memory a 

7 respective interpolated on-grid brightness value at the received pixel position; and 
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8 (iii) a response function compensator for compensating the respective interpolated 

9 on-grid brightness value derived by the interpolator at each pixel position received from the pixel 

10 position generator by the inverse of a combined response function of the light-sensing elements 

1 1 to derive a respective resampled on-grid brightness value at each pixel position received from the 

12 pixel position generator, the combined response function of the light-sensing elements being the 

13 sum of weighted response functions of light-sensing elements having different ones of the 

14 sensitivity levels, each one of the weighted response functions being a response function of a 
1:15 light-sensing element having a respective one of the sensitivity levels multiplied by a 

i:l6 predetermined weighting factor for the respective one of the sensitivity levels. 

*'?^1 93. The system of claim 92, wherein the response function compensator comprises a 

i;32 lookup table memory storing data representing the inverse of the combined response function of 

i 03 the light-sensing elements and a mapper for deriving from the respective interpolated on-grid 

I:f4 brightness value derived by the interpolator at a received pixel position a respective index for the 

5 lookup table memory and providing a data value in the lookup table memory corresponding to 

6 the respective index to the output image memory, the data value being the pixel brightness value 

7 of the output image at the received pixel position. 

1 94. The system of claim 91 , wherein the accumulator comprises an adder for adding the 

2 sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid 
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3 brightness value estimator for corresponding sensitivity levels at each received off-grid position 

4 to derive the respective estimated off-grid brightness value at each off-grid positions received 

5 from the off-grid position generator. 

1 95. The system of claim 91 , wherein the accumulator comprises a weighting factor 

2 memory for storing a plurality of predetermined weighting factors each corresponding to a 

3 different one of the plurality of sensitivity levels of the light-sensing elements, a plurality of 
multipliers each for multiplying the sensitivity level off-grid brightness value corresponding to a 

□15 respective one of the sensitivity levels at a received off-grid position by the weighting factor 
corresponding to the respective one of the sensitivity levels to provide a respective weighted 

"^7 sensitivity level off-grid brightness value for the corresponding sensitivity level at the received 

□8 off-grid position, and an adder for summing the respective weighted sensitivity level off-grid 

fIB brightness values provided by the plurality of multipliers to derive a respective estimated off-grid 

=:;f 0 brightness value at the received off-grid position received from the off-grid position generator. 

1 96. The system of claim 77, wherein the off-grid estimator comprises: 

2 (a) an off-grid position generator for providing the off-grid positions of the off-grid 

3 position array; 

4 (b) a sensitivity pattern memory for storing data indicative of the sensitivity level 

5 corresponding to each one of the pixel positions of the pixel position array; 
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6 (c) a plurality of sensitivity level off-grid brightness value estimators, each 

7 corresponding to a respective one of the sensitivity levels, for receiving off-grid positions from 

8 the off-grid position generator and sensitivity pattern data from the sensitivity pattern memory, 

9 each one of the plurality of sensitivity level off-grid brightness value estimators being responsive 

10 to an off-grid position received ftom the off-grid position generator and the sensitivity pattern 

1 1 data from the sensitivity pattern memory for deriving from the captured image brightness values 

12 in the captured image memory a respective sensitivity level off-grid brightness value for the 
[|3 corresponding sensitivity level at the received off-grid position; 

\M (d) an accumulator for combining the respective sensitivity level off-grid brightness 

•' J5 values derived by the plurality of sensitivity level off-grid brightness value estimator for the 

■^6 corresponding sensitivity levels at each off-grid position received from the off-grid position 

i; J7 generator to derive a respective combined sensitivity level off-grid brightness value at each off- 

i'iS grid position received from the off-grid position generator; and 
9 (e) a response function compensator for compensating the respective combined sensitivity 

20 level off-grid brightness value at each off-grid position received from the off-grid position 

21 generator by the inverse of a combined response function of the light-sensing elements to derive 

22 a respective estimated off-grid brightness value at each off-grid position received from the off- 

23 grid position generator, the combined response function being the sum of weighted response 

24 functions of light-sensing elements having different ones of the plurality of sensitivity levels, 

25 each one of the weighted response functions being a response function of a light-sensing element 
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26 having a respective one of the sensitivity levels multiplied by a predetermined weighting factor 

27 for the respective one of the sensitivity levels. 

1 97. The system of claim 96, wherein the response function compensator comprises a 

2 lookup table memory storing lookup table data representing the inverse of the combined response 

3 function of the light-sensing elements^ and a mapper for deriving from the respective combined 

4 sensitivity level off-grid brightness value at each off-grid position received from the off-grid 
□;5 position generator a respective index for the lookup table memory and providing a data value in 

the lookup table memory corresponding to the respective index as the respective estimated off- 

■'^7 grid brightness value at the off-grid position received from the off-grid position generator. 

CjI 98. The system of claim 96, wherein the output image generator comprises: 

i 92 (a) an off-grid brightness value memory for storing the respective estimated off-grid 

1;!: 3 brightness value derived by the response function compensator at each off-grid position received 

4 from the off-grid position generator; 

5 (b) an on-grid resampler for deriving from the estimated off-grid brightness values in the 

6 off-grid brightness value memory respective resampled on-grid brightness values at the pixel 

7 positions of the pixel position array; and 

8 (c) an output image memory for storing the respective resampled on-grid brightness 

9 values derived by the on-grid resampler as the pixel brightness values of the output image, 
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1 0 and wherein the on-grid resampler comprises: 

11 (i) a pixel position generator for providing pixel positions of the pixel position 

12 array; and 

13 (ii) an interpolator for receiving pixel positions provided by the pixel position 

14 generator and being responsive to a pixel position received from the pixel position generator for 

15 deriving from the estimated off-grid brightness values in the off-grid brightness value memory a 

16 respective resampled on-grid brightness value at the received pixel position. 

L;ril 99. The system of claim 96, v^herein the accumulator of the off-grid estimator comprises 

an adder for adding the respective sensitivity level off-grid brightness values derived by the 
plurality of sensitivity level off-grid brightness value estimators for the corresponding sensitivity 
levels at each off-grid position received from the off-grid position generator to provide the 

[lJ5 combined sensitivity level off-grid brightness values at each off-grid position received from the 

:;:f6 off-grid position generator. 

1 100. The system of claim 96, wherein the accumulator of the off-grid estimator 

2 comprises a weighting factor memory for storing a plurality of predetermined weighting factors 

3 each corresponding to a different one of the plurality of sensitivity levels of the light-sensing 

4 elements, a plurality of multipliers each for multiplying the sensitivity level off-grid brightness 

5 value corresponding to a respective one of the sensitivity levels at a received off-grid position by 
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6 the weighting factor corresponding to the respective one of the sensitivity levels to provide a 

7 weighted sensitivity level off-grid brightness value for the corresponding sensitivity level at the 

8 off-grid position received from the off-grid position generator, and an adder for summing the 

9 weighted sensitivity level off-grid brightness values at the received off-grid position provided by 

10 the plurality of multipliers to derive the combined sensitivity level off-grid brightness value at 

11 the off-grid positions received from the off-grid position generator. 

gl 101 . The system of claim 100, wherein the array of light-sensing elements of the image 

sensor is a linear array and the pixel position array is a linear array having a multiplicity of pixel 
positions at respective regularly spaced pixel positions in a pixel row, and wherein the off-grid 
positions of the off-grid position array are each located between a respective adjacent pair of 
i;;::=5 pixel positions of the pixel position array and the sensitivity level off-grid brightness values 
! U6 derived by the plurality of sensitivity level off-grid brightness value estimators for an off-grid 
;;:f7 position received from the off-grid position generator are respective captured image brightness 

8 values at a predefined number of pixel positions that are nearest neighbors to the received off- 

9 grid position, each one of the predefined number of nearest neighbor pixel positions 

10 corresponding to a different one of the plurality of sensitivity levels of the light-sensing 

1 1 elements. 
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1 102. The system of claim 101, wherein each one of the off-grid positions of the off-grid 

2 position array is located midway between a respective adjacent pair of pixel positions of the 

3 linear pixel position array and the predetermined sensitivity pattern of the light-sensing elements 

4 has a first and a second predetermined sensitivity level, and wherein the combined sensitivity 

5 level off-grid brightness value at an off-grid position x' received from the off-grid position 

6 generator is expressed as 

7 ^(x') = W^I^ix' - 0.5) + W^Ipc' + 0.5), 

£38 where B(x') is the combined sensitivity level off-grid brightness value at the received off-grid 

Ul9 position x', is the weighting factor for the sensitivity level corresponding to pixel position 

^'f 0 (x' - 0.5), W2 is the weighting factor for the sensitivity level corresponding to pixel position 

^'IJl (x' + 0.5), Ic(x' - 0.5) is the captured image brightness value at pixel position (x' - 0.5) and 

Ic(x' + 0.5) is the captured image brightness value at pixel position (x' + 0.5), and the combined 

I Il3 response function of the light-sensing elements is expressed as 
;;|4 S{E) - W^P^iE) ^ W/^(E), 

15 where Pi(E) is the radiometric response function of a light-sensing element having the sensitivity 

16 level corresponding to pixel position (x' - 0.5) and PsCE) is the radiometric response function of 

17 a light-sensing element having the sensitivity level corresponding to pixel position (x' + 0.5). 

1 103. The system of claim 100, wherein the image sensor has a two-dimensional array of 

2 light-sensing elements and the multiplicity of pixel positions are located at respective 
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3 intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel 

4 columns defining a two-dimensional pixel position array, the pixel rows being orthogonal to the 

5 pixel columns, the multiplicity of off-grid positions are located at respective intersections of a 

6 plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate 

7 columns, each one of the intermediate rows being parallel to the pixel rows and extending 

8 between a respective adjacent pair of the pixel rows, each one of the intermediate columns being 

9 parallel to the pixel columns and extending between a respective adjacent pair of the pixel 
i;|0 columns. 

^'Jll 104. The system of claim 103, wherein the predetermined sensitivity pattern of the array 

r.-;2 of light-sensing elements corresponds to a pixel position array having repetitive disposed groups 

of four nearest neighbor pixel positions, where each one of the four nearest neighbor pixel 

ry4 positions in each group corresponds to a respective one of four predetermined sensitivity levels 

::S5 in a common predetermined positional order. 

1 105. The system of claim 100, wherein the image sensor has a two-dimensional array of 

2 light-sensing elements and the multiplicity of pixel positions are located at respective 

3 intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel 

4 columns defining a two-dimensional pixel position array, the pixel rows being orthogonal to the 

5 pixel columns, and wherein the multiplicity of off-grid positions are located at respective 
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6 intersections of a plurality of regularly spaced intermediate rows and the pixel columns, and at 

7 respective intersections of a plurality of regularly spaced intermediate columns and the pixel 

8 rows, each one of the intermediate rows being parallel to the pixel rows and extending between a 

9 respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to 
10 the pixel columns and extending between a respective adjacent pair of the pixel columns. 

1 106. The system of claim 100, wherein the predetermined sensitivity pattern of the array 

i-^Z of light-sensing elements has a first and a second predetermined sensitivity level, and adjacent 

iJQ pixel positions in each one of the pixel rows and adjacent pixel positions in each one of the pixel 

^"^4 columns correspond to different ones of the predetermined first and second sensitivity levels. 

rll 1 07. The system of claim 1 00, wherein the array of light-sensing elements of the image 

fill sensor is a linear array and the multiplicity of pixel positions are located at respective regularly 

'::?3 spaced pixel positions in a pixel row so as to form a linear captured image brightness value array, 

4 and wherein the linear array of light-sensing elements of the image sensor is adapted to be 

5 sequentially exposed to successive regularly spaced linear regions of the scene in multiple 

6 exposures, each one of the exposures producing a respective linear captured image brightness 

7 value array, the respective linear captured image brightness value arrays produced by the 

8 multiple exposures being concatenated to form a two-dimensional captured image brightness 

9 value array having respective captured image brightness values at pixel positions of a pixel 
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10 position array located at respective intersections of a plurality of regularly spaced pixel rows and 

1 1 a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel 

12 columns, and wherein the multiplicity of off-grid positions are located at respective intersections 

13 of the pixel rows and a plurality of regularly spaced intermediate columns, each one of the 

14 intermediate columns being parallel to the pixel colunms and extending between a respective 

1 5 adj acent pair of the pixel columns. 

q:1 108. The system of claim 107, wherein the predetermined sensitivity pattern of the array 

102 of light-sensing elements has a first and a second predetermined sensitivity level, and wherein 

^'3 adjacent pixel positions in each pixel row correspond to different ones of the first and the second 

'^li^ predetermined sensitivity levels and adjacent pixel positions in each pixel column correspond to 

□5 the same one of the first and the second predetermined sensitivity levels. 

;::f 1 109. The apparatus of claim 91 or 96, wherein each one of the sensitivity level off-grid 

2 brightness value estimators is responsive to an off-grid position received from the off-grid 

3 position generator and the data indicative of the sensitivity level corresponding to each one of the 

4 pixel positions from the sensitivity pattem memory for determining a pixel position 

5 corresponding to the same sensitivity level as the sensitivity level off-grid brightness value 

6 estimator and located nearest the received off-grid position, and for estimating a sensitivity level 

7 off-grid brightness value at the received off-grid position to be equal to the captured image 
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8 brightness value at the nearest pixel position corresponding to the same sensitivity level as the 

9 sensitivity level off-grid brightness value estimator. 

1 110. The method of claim 92, wherein the interpolator is responsive to a pixel position 

2 received from the pixel position generator for computing a product of the estimated off-grid 

3 brightness values at a predetermined number of off-grid positions that are nearest neighbors to 

4 the received pixel position and an interpolation kernel having the same dimensions as the 
rn5 predetermined number of off-grid positions. 

^'f 1 111. The apparatus of claim 92, wherein the pixel positions provided by the pixel 

"~2 position generator are located at respective intersections of a plurality of regularly spaced pixel 

[33 rows and a plurality of regularly spaced pixel columns, the pixel columns being orthogonal to the 

rU4 pixel rows, and the off-grid positions provided by the off-grid position generator are located at 

;;:f 5 respective intersections of a plurality of regularly spaced intermediate rows and a plurality of 

6 regularly spaced intermediate columns, each one of the intermediate rows being parallel to the 

7 pixel rows and extending medially between a respective adjacent pair of the pixel rows, each 

8 intermediate column being parallel to the pixel columns and extending medially between a 

9 respective adjacent pair of the pixel columns, and wherein the interpolator derives the 

10 interpolated on-grid brightness values at each one of the pixel positions (x,y) received from the 

1 1 pixel position generator using the relation 
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i=o j=0 

12 where x has values from 25 to xSize -2.5, y has values from 2.5 to ySize -2.5, 1(x"2, y-2) is 

13 the interpolated on-grid brightness value at pixel position (x-2, y-2), B(x- 1 .5+i, y- 1 .5+j) is the 

14 off-grid brightness value at off-grid position (x- 1 .5+i, y- 1 .5+j), (j)y is the value of a 4 x 4 cubic 

1 5 interpolation kernel 4) at indices i and j, indices i and j each has values from 0 to 3, xSize is the 
:^|6 dimension of the captured image in the direction of the pixel rows, and ySize is the dimension of 
W the captured image in the direction of the pixel columns. 

'j^ 1 12. The apparatus of claim 98, wherein the pixel positions provided by the pixel 

t!^ position generator are located at respective intersections of a plurality of regularly spaced pixel 

ry3 rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the 

'vfA pixel columns, and the off-grid positions provided by the off-grid position generator are located 

5 at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of 

6 regularly spaced intermediate columns, each one of the intermediate rows being parallel to the 

7 pixel rows and extending medially between a respective adjacent pair of the pixel rows, each one 

8 of the intermediate columns being parallel to the pixel columns and extending medially between 

9 a respective adjacent pair of the pixel columns, and wherein the interpolator derives the 
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10 resampled on-grid brightness values at each one of the pixel positions (x,y) received from the 

1 1 pixel position generator by the relation 

12 where x has values from 2.5 to xSize-2.5, y has values from 2.5 to ySize-2.5, 1(x-2, y-2) is the 

1 3 resampled on-grid brightness value at pixel position (x-2, y-2), B(x- 1 .5+i,y- 1 .5+j) is the off- 
r|4 grid brightness value at off-grid position (x- 1 .5+i, y- 1 .5+j), cl)y is the value of a 4 x 4 cubic 
Ln5 interpolation kernel (J) at indices i and the indices i and j each have values from 0 to 3, xSize is 

6 the dimension of the captured image in the direction of the pixel rov^s, and ySize is the 

f ^7 dimension of the captured image in the direction of the pixel columns. 

rUl 113. The apparatus of claim 92, wherein the pixel positions provided by the pixel 

'v£2 position generator are located at respective intersections of a plurality of regularly spaced pixel 

3 columns and a pluraity of regularly spaced pixel rows, the pixel rows being orthogonal to the 

4 pixel columns, and the off-grid positions provided by the off-grid generator are located at 

5 respective intersections of a plurality of regularly spaced intermediate rows and the pixel 

6 columns and at respective intersections of a plurality of regularly spaced intermediate columns 

7 and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and 

8 extending medially between a respective adjacent pair of the pixel rows, each one of the 
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9 intermediate columns being parallel to the pixel columns and extending medially between a 

1 0 respective adjacent pair of the pixel columns, and wherein the interpolator derives the 

1 1 interpolated on-grid brightness values at each one of the pixel positions (xj) received from the 

1 2 pixel position generator using the relation 

3 3 

/(x-2j;-2) - E E 5(x+0.5/-0.5/,>^-L5+0,5/+0.5/)(i) , 

i:|3 where x has values from 2.5 to xSize -2.5, y has values from 2.5 to ySize -2.5, 1(x-2, y-2) is 

iJ4 the interpolated on-grid brightness value at pixel position (x~2, y-2), 

^'f 5 B(x+0.5i"0.5j, y- 1 .5+0.5i+0.5j) is the off-grid brightness value at off-grid position 

|'j£6 (x+0.5i"0.5j, y- 1 .5+0.5i+0.5j), is the value of a 4 x 4 cubic interpolation kernel (|) at indices i 

:;|7 and j, indices i and j each have values from 0 to 3, xSize is the dimension of the captured image 

riS in the direction of the pixel rows, and ySize is the dimension of the captured image in the 

^;:i9 direction of the pixel columns. 

1 114. The apparatus of claim 98, wherein the pixel positions provided by the pixel 

2 position generator are located at respective intersections of a plurality of regularly spaced pixel 

3 rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the 

4 pixel columns, and the off-grid positions provided by the off-grid position generator are located 

5 at respective intersections of a plurality of regularly spaced intermediate rows and the pixel 
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6 colximns and at respective intersections of a plurality of regularly spaced intermediate columns 

7 and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and 

8 extending medially between a respective adjacent pair of the pixel rows, each one of the 

9 intermediate columns being parallel to the pixel columns and extending medially between a 

10 respective adjacent pair of the pixel columns, and wherein the interpolator derives the resampled 

1 1 on-grid brightness values at each one of the pixel positions (x,y) received from the pixel position 

1 2 generator using the relation 

5 I(x~2^~2) = E E 5(x+0.5/-0.5/, j;-1.5+0.5/-H0.5/)(t). 

Ill i-o j=0 ^ 

'^^3 where x has values from 2.5 to xSize-2.5, y has values from 2.5 to ySize-2.5, 1(x-2,y-2) is the 

f|4 resampled on-grid brightness value at pixel position (x-2,y-2), B(x+0.5i-0.5j, y- 1 .5+0.5i+0.5j) 

ills is the off-grid brightness value at the off-grid position (x+0.5i-0,5j, y- 1 .5+0.5i+0.5j), (|)y is the 

^:|6 value of a 4 x 4 cubic interpolation kernel (}) at indices i and j, indices i and j each have values 

17 from 0 to 3, xSize is the dimension of the captured image in the direction of the pixel rows, and 

18 ySize is dimension of the captured image in the direction of the pixel columns. 
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115. The method of claim 1 1 1 or 1 12, wherein the 4 x 4 cubic interpolation kemel (j) has 
the form: 
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116. The system of claim 98, wherein the pixel positions provided by the pixel position 
generator are located at respective regularly spaced pixel positions in a pixel row defining a 
linear pixel position array and the off-grid positions provided by the off-grid position generator 
are each located midway between a respective adjacent pair of pixel positions of the pixel 
position row, and wherein the interpolator derives the resampled on-grid brightness values at 
each one of the pixel positions x received from the pixel position generator using the relation 

A- 1.5 

where x has values from 3.5 to xSize-3.5, Io(x - 3) is the resampled on-grid brightness value at 
pixel position (x - 3), B(x + k) is the estimated off-grid brightness value at off-grid position 
(x + k), k is a position of a 4 x 1 cubic interpolation kemel i};, ijf (k) is the value of the cubic 
interpolation kemel i|r at position k, the position k has values from - 1 .5 to 1 .5, and xSize is the 
dimension of the captured line image. 
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1 117. The system of claim 116, wherein the 4 x l cubic interpolation kernel x has the 

2 form: 



-0.96 


13.44 


13.44 


-0.96 
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5 k 

1 118. The apparatus of claim 77, wherein the image sensor is an integrated circuit charge 

2 coupled device having an array of photodiodes in spaced orthogonal rows and columns, each one 
^^3 of the photodiodes having a light-sensing surface adapted to be exposed to incident light and 
fS4 being operatively biased to store photogenerated charge therein, the charge coupled device 

l-:^ having respective integrated lenses formed over the photosensitive surfaces of predetermined 

-n6 ones of the photodiodes, each integrated lens concentrating the incident light onto the respective 

'^JJ light-sensing surface, whereby the sensitivity level of each one of the photodiodes is determined 

rfS by whether or not an integrated lens is formed over the light-sensing surface thereof. 

1 119. The apparatus of claim 77, wherein the image sensor is an integrated circuit charge 

2 coupled device having an array of photodiodes arranged in spaced orthogonal rows and columns, 

3 each one of the photodiodes having a light-sensing surface adapted to be exposed to incident 

4 light and being operatively biased to store photogenerated charge therein, each one of the 

5 photodiodes having an integrated light filter formed above the light-sensing surface thereof, the 

6 transparency of each integrated light filter being fixed during fabrication of the charge coupled 
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7 device, whereby the sensitivity level of each one of the photodiodes in the array is determined by 

8 the transparency of the integrated light filter above the light-sensing surface thereof. 

1 120. The system of claim 77, wherein the image sensor is an integrated circuit charge 

2 coupled device having an array of photodiodes arranged in spaced orthogonal rows and columns, 

3 each one of the photodiodes having a light-sensing surface adapted to be exposed to incident 

4 light and being operatively biased to store photogenerated charge therein, the charge coupled 
o5 device being formed with an opaque layer overlying the array of photodiodes with a respective 
yl6 aperture formed in the opaque layer above the light-sensing surface of each one of the 

="!7 photodiodes, the respective size of the aperture above the light-sensing surface of each one of the 

"J=J8 photodiodes being fixed during fabrication of the charge coupled device, whereby the sensitivity 

cfi level of each one of the photodiodes in the array is determined by the size of the aperture above 

rUO the light-sensing surface thereof. 

1 121 . The system of claim 77, wherein the image sensor is an integrated circuit charge 

2 coupled device having a multiplicity of photodiodes in an array of spaced orthogonal rows and 

3 columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to 

4 incident light and being operatively biased to store photogenerated charge therein, the charge 

5 coupled device having an interline structure in which the columns of photodiodes are each 

6 disposed adjacent a corresponding vertical shift register, each photodiode having an associated 
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7 transfer gate electrode for controlling the transfer of stored photogenerated charge from the 

8 photodiode to the corresponding vertical shift register, the stored photogenerated charge in each 

9 one of the photodiodes being operatively periodically removed by the application of a periodic 

10 reset pulse to the device, the photogenerated charge stored in each one of the photodiodes being 

1 1 operatively transferred to a respective stage of the corresponding vertical shift register by the 

12 application of a transfer trigger pulse to the associated transfer gate electrode and being 

13 accumulated therein for an exposure interval, the transfer trigger pulse being applied immediately 
r|4 preceding a reset pulse, the respective transfer trigger pulses being operatively applied to the 

115 transfer gate electrodes associated with the photodiodes in the array at predetermined 

"t6 frequencies, whereby the sensitivity level of each one of the photodiodes in the array is 

^117 determined by the frequency of transfer trigger pulses applied to the transfer gate electrode 

I- Ji8 associated therewith. 

::;f 1 122. The system of claim 77, wherein the off-grid estimator comprises: 

2 (a) a pixel position generator for providing the pixel positions of the pixel position array; 

3 (b) a sensitivity pattern memory for storing the sensitivity level corresponding to each 

4 one of the pixel positions; 

5 (c) a response fimction compensator receiving pixel positions of the pixel position array 

6 from the pixel position generator, data indicative of the sensitivity levels corresponding to the 

7 received pixel positions from the sensitivity pattern memory and captured image brightness 
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8 values at the received pixel positions from the captured image memory, and being responsive to a 

9 pixel position received from the pixel position generator and data indicative of the sensitivity 

10 level corresponding to the received pixel position for retrieving from the captured image 

1 1 memory the captured image brightness value at the received pixel position and for compensating 

12 the retrieved captured imaged brightness value at the received pixel positions by the inverse of a 

13 response ftmction of a light-sensing element having the sensitivity level corresponding to the 

14 received pixel position to provide a respective compensated brightness value at the pixel position 
;i5 received from the pixel position generator; 

lfl6 (d) a compensated on-grid brightness value memory for storing respective compensated 

^'f7 captured image brightness values derived by the response ftmction compensator; 

(e) an off-grid position generator for providing the off-grid positions of the off-grid 
position array; 

two (f) a pseudoinverse estimator receiving off-grid positions from the off-grid position 

i=il generator and being responsive to an off-grid position received from the off-grid position 

22 generator for deriving from the compensated captured image brightness values in the 

23 compensated on-grid brightness value memory a respective estimated off-grid brightness value at 

24 the off-grid positions received from the off-grid position generator. 

1 123. The system of claim 122, wherein the output image generator comprises 
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2 (i) an off-grid brightness value memory for storing the respective estimated off- 

3 grid brightness values derived by the pseudoinverse estimator; 

4 (ii) an on-grid resampler for deriving from the estimated off-grid brightness 

5 values in the off-grid brightness value memory respective on-grid resampled brightness values at 

6 the pixel positions of the pixel position array; and 

7 (iii) an output image memory for storing the respective resampled on-grid 

8 brightness values derived by the on-grid resampler as pixel brightness values of the output 
rj9 image, and wherein the on-grid resampler comprises: 

liO (1) a second pixel position generator for providing the pixel positions of 

^'|l the pixel position array; and 

^1^2 (2) an interpolator receiving pixel positions from the pixel position 

r |3 generator and being responsive to a pixel position received from the second pixel position 

ri4 generator for deriving from the estimated off-grid brightness values in the off-grid brightness 

|:i5 value memory the resampled on-grid brightness value at the pixel position received from the 

1 6 second pixel position generator. 

1 124. The system of claim 122, wherein the response fimction compensator comprises a 

2 lookup table memory storing separate lookup table data representing the inverse of respective 

3 response functions of light-sensing elements having different ones of the plurality of sensitivity 

4 levels, and a mapper receiving from the captured image memory the captured image brightness 
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5 value at the pixel position received from the first pixel position generator for deriving therefrom 

6 a respective index for the lookup table data representing the inverse of the response function of a 

7 light-sensing element having the sensitivity level corresponding to the received pixel position^ 

8 and providing a compensated on-grid brightness value corresponding to the index to the 

9 compensated on-grid brightness value memory. 

1 125. The system of claim 122, wherein the pseudoinverse estimator comprises means 

r-2 responsive to an off-grid positions received from the off-grid position generator for retrieving 

\jB from the compensated on-grid brightness value memory respective compensated on-grid 

brightness values at a predetermined number of pixel positions that are nearest neighbors to the 

'";^5 received off-grid position, a comparator for comparing each one of the respective compensated 

j;36 on-grid brightness values at the nearest neighbor pixel positions to a predetermined low threshold 

fW value and a predetermined high threshold value, an adder means for computing a sum of 

|;:f8 compensated on-grid brightness values at the nearest neighbor pixel positions that are greater 

9 than the predetermined low threshold value and less than the predetermined high threshold value, 

10 and a divider for deriving the estimated off-grid brightness value at the received off-grid position 

1 1 by dividing the sum computed by the adder by the number of compensated on-grid brightness 

12 values included in the sum. 
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1 126. The system according to claim 122, wherein the pixel positions provided by the 

2 pixel position generator are located at respective intersections of a plurality of regularly spaced 

3 pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to 

4 the pixel columns, and the off-grid positions provided by the off-grid position generator are 

5 located at respective intersections of a plurality of regularly spaced intermediate rows and a 

6 plurality of regularly spaced intermediate columns, each one of the intermediate rows being 

7 parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel 
rjS rows, each intermediate column being parallel to the pixel columns and extending medially 

1.09 between a respective adjacent pair of pixel columns, and wherein the off-grid positions provided 

^'Jo by the off-grid position generator have coordinates (x',y ') where x' has values from 1 to xSize- 1 

jil and y' has values from 1 to y Size- 1, xSize being the dimension of the captured image in the 

j;;12 direction of the pixel rows and ySize being the dimension of the captured image in the direction 

113 of the pixel columns. 

1 127. The system of claim 126, wherein the predetermined sensitivity pattern of the array 

2 of light-sensing elements corresponds to a pixel position array having repetitively disposed 

3 groups of four nearest neighbor pixel positions, where each of the four nearest neighbor pixel 

4 positions in each group corresponds to a respective one of four predetermined sensitivity levels 

5 in a common predetermined positional order. 
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128, The system of claim 126, wherein the pseudoinverse estimator for estimating the 
respective brightness value at an off-grid position (x',y') received from the off-grid position 
generator comprises: 

(i) a memory for storing a 4 x 4 cubic interpolator kemel (|); 

(ii) a comparator for comparing the compensated on-grid brightness value at each 
one of sixteen pixel positions (u,v) that are nearest neighbors to the received off-grid position 
(x',y ') with a predetermined low threshold value and a predetermined high threshold value; 

(iii) first computing means for calculating the indices i and j of the 4 x 4 cubic 
interpolation kemel (|) using the relations i = x'-u- 1.5 and j = y'-v- 1.5 for each one of the 
sixteen nearest neighbor pixel positions (u,v) having a compensated on-grid brightness value 
I(u,v) that is greater than the predetermined low threshold value and less than the predetermined 
high threshold value, where u has values from x'-1.5tox'+L5 and v has values from y' - 1.5 to 

y'+l.5; 

(iv) second computing means for calculating the quantity 

^ (j) I(u,v) 

ab 

for each compensated on-grid brightness value I(u,v) at the sixteen nearest neighbor pixel 
positions (u,v) that is greater than the predetermined threshold value and less than the 
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17 predetermined threshold value using the indices i and j calculated by the first computing means, 

1 8 where <^^^ is the value of the 4 x 4 interpolating kernel ^ at indices m and n; 

1 9 (v) third computing means for calculating the quantities (})^., for values of i and j 

20 calculated by the first computing means; 

21 (vi) first adder for adding the quantities 

ab 

^ J2 calculated by the second computing means to derive a first sxjm s; 

^'|3 (vii) second adder for adding the quantities 4)^. calculated by the third computing 

■ '34 means to derive a second sum w; and 

:::25 (viii) a divider for deriving the estimated off-grid brightness value at the received 

rM6 off-grid position (x\y by dividing the first sum s by the second sum w. 

1 129. The system according to claim 122, wherein the pixel positions provided by the 

2 pixel position generator are located at respective intersections of a plurality of regularly spaced 

3 pixel rows and a plurality of regularly spaced pixel columns, and the off-grid positions provided 

4 by the off-grid position generator are located at respective intersections of a plurality of regularly 

5 spaced intermediate rows and the pixel columns, and at respective intersections of a plurality of 

6 regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows 
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7 being parallel to the pixel rows and extending medially between a respective adjacent pair of the 

8 pixel rows, each intermediate column being parallel to the pixel columns and extending medially 

9 between a respective adjacent pair of the pixel columns, the off-grid positions provided by the 

10 off-grid position generator having coordinates (x'y') where x' has values from 1 to xSize- 1 and 

11 y' has values from 0.5 to ySize-0.5 for off-grid positions located at respective intersections of 

12 the pixel rows and the intermediate columns, and where x' has values from 0.5 to ySize-0.5 and 

13 y' has values from 1 to ySize- 1 for off-grid positions located at respective intersections of the 
pixel columns and the intermediate rows, xSize being the dimension of the captured image in the 

lis direction of the pixel rows and ySize being the dimension of the captured image in the direction 
of the pixel columns. 

ril 130. The system of claim 129, wherein the predetermined sensitivity pattem of the array 

rLS of light- sensing elements has first and second predetermined sensitivity levels, and adjacent pixel 
^--3 positions in each pixel row and adjacent pixel positions in each pixel column correspond to 
4 different ones of the first and second predetermined sensitivity levels. 

1 131. The system of claim 129, wherein the pseudoinverse estimator for estimating the 

2 respective off-grid brightness value at an off-grid position (x',y') received from the off-grid 

3 position generator comprises; 

4 (i) a memory for storing a 4 x 4 cubic interpolation kernel (|); 



NY02:241764-1 



-154- 



FILE NO. A32708-070050.1125 



5 (ii) first computing means responsive to an off-grid position (x',y ') received from the off- 

6 grid position generator for computing coordinates (u^v) using the relations u=x'+0.5 i-0.5j and 

7 v=y'- 1 .5+0.5i+0.5j, where i and j are indices of the 4 x 4 cubic interpolation kernel (|), and each 

8 one of the indices i and j has values from 0 to 3; 

9 (iii) means responsive to each pair of coordinates (u,v) computed by the first computing 

10 means for determining whether the coordinates (u^v) are those of a pixel position; 

1 1 (iv) a comparator responsive to the coordinates (u,v) being coordinates of a pixel position 
fi^ for comparing the compensated on-grid brightness value I(u,v) at the pixel position (u,v) with a 
[JFiS predetermined low threshold value and a predetermined high threshold value; 

\'f4 (v) second computing means responsive to the coordinates (u,v) being the coordinates of 

|1;5 a pixel position and the compensated on-grid brightness value I(u,v) at the pixel position (u,v) 

r|6 being greater than a predetermined low threshold value and less than a predetermined high 

r|7 threshold value for calculating the quantity 

(1)2 

ab 

18 where the values of i and j are those used by the first computing means to calculate the 

19 coordinate (u,v), and is the value of the 4 x 4 cubic interpolation kernel ^ at indices m and n; 

20 (vi) third computing means for calculating the quantity (|)^ for values of i and j used by 

21 the first computing means to calculate pixel position coordinates (u,v) at which the compensated 
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22 on-grid brightness value I(u,v) is greater than the predetermined low threshold value and less 

23 than the predetermined high threshold value; 

24 (vii) a first adder for adding the quantities 

, (j) I(u,v) 
? 

ab 

25 calculated by the second computing means to derive a first sum s; 

r|6 (viii) a second adder for adding the quantities (j)^. calculated by the third computing 

107 means to derive a second sum w; and 

^'28 (ix) a divider for deriving the estimated off-grid brightness value at the off-grid position 

^29 (x',y') received fi*om the off-grid position generator by dividing the first sum s by the second 

r30 sumw. 

'vf 1 132. The system of claim 131, wherein the 4 x 4 cubic interpolation kernel cj) has the 

2 form: 
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1 133. The system of claim 122, wherein the pixel positions provided by the pixel position 

2 generator are located at respective regularly spaced pixel positions in a linear pixel position array 

3 and the off-grid positions provided by the off-grid position generator are each located midway 

4 between a respective adjacent pair of pixel positions of the linear pixel position array, and 

5 wherein the predetermined sensitivity pattem of the light-sensing elements has a first and a 

6 second predetermined sensitivity level, and adjacent pixel positions of the linear pixel position 

7 array correspond to different ones of the first and second predetermined sensitivity levels. 
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1 134. The system of claim 133, wherein the pixel positions provided by the pixel position 

2 generator has values from 0.5 to xSize-0,5, where xSize is the dimension of the captured line 

3 image, and the off-grid positions provided by the off-grid position generator have values from 2 

4 toxSize-2. 

1 135. The system of claim 134, wherein the pseudoinverse estimator for estimating the 

2 respective brightness values at an off-grid position x' received from the off-grid position 
;.=:3 generator comprises: 

i f4 (i) a memory for storing a 4 x 1 Gaussian interpolation kernel G; 

h-5 (ii) a comparator for comparing the compensated on-grid brightness value I(k) at each one 

' Jj6 of four pixel positions k that are nearest neighbors to the received off-grid position x' with a 

predetermined low threshold value and a predetermined high threshold value, where k has values 

ry8 fromx'-1.5tox'+1.5; 

^"j9 (iii) first computing means for calculating the quantity I(k)G(k-x') for each one of the 

10 compensated on-grid brightness values I(k) at the four nearest neighbor pixel positions k that is 

1 1 greater than the predetermined low threshold value and less than the predetermined high theshold 

12 value, where G(k-x') is the value of the Gaussian interpolation kernel G at position (k-x'); 

13 (iv) a first adder for adding the quantities I(k)G(k-x') for all values of I(k) greater than 

14 the predetermined low threshold value and less than the predetermined high threshold value to 

1 5 derive a first sum p; 
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16 (v) a second adder for adding the Gaussian interpolation kernel values G(k-x') for all 

1 7 values of k v^here I(k) is greater than the predetermined low threshold value and less than the 

1 8 predetermined high threshold value to derive a second sum q; and 

19 (vi) a divider for deriving the estimated off-grid brightness value B(x') at the off-grid 

20 position x' received from the off-grid position generator. 

1 136. The apparatus of claim 125, 128, 131 or 135, wherein the predetermined low 

threshold value is the noise level of a light-sensing element having a sensitivity level 

S3 corresponding to the pixel position of the compensated on-grid brightness value being compared 

^""4 therewith compensated by the inverse of the response function of the light-sensing element, and 

|i;5 the predetermined high threshold value is the saturation or near saturation brightness value of a 

f l6 light-sensing element having the sensitivity level corresponding to the pixel position of the 

rU7 compensated on-grid brightness value being compared therewith. 

1 137. The apparatus of claim 123, wherein the pixel positions of the pixel position array 

2 provided by the second pixel position generator are located at respective regularly spaced pixel 

3 positions of a linear pixel position array, and the off-grid positions of the off-grid position array 

4 provided by the off-grid position generator are each located between a respective adjacent pair of 

5 pixel positions of the linear pixel position array. 
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1 138. The apparatus of claim 123, wherein the pixel positions of the pixel position array 

2 provided by the second pixel position generator are located at respective intersections of a 

3 plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the 

4 pixel rows being orthogonal to the pixel columns, and wherein the off-grid positions of the off- 

5 grid position array provided by the off-grid position generator are located at respective 

6 intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly 

7 spaced intermediate coliimns, each intermediate row being parallel to the pixel rows and 

,-.^8 extending between a respective adjacent pair of the pixel rows and each intermediate column 

\f9 being parallel to the pixel columns and extending between a respective adjacent pair of the pixel 

i^fO columns. 

l^^-l 139. The apparatus of claim 123, wherein the pixel positions of the pixel position array 

jij2 provided by the second pixel position generator are located at respective intersections of a 

533 plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the 

4 pixel rows being orthogonal to the pixel columns, and wherein the off-grid positions of the off- 

5 grid position array provided by the off-grid position generator are located at respective 

6 intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one 

7 of the intermediate pixel columns being parallel to the pixel columns and extending between a 

8 respective adjacent pair of pixel colunms. 
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1 140. The apparatus of claim 137, 138 or 139, wherein the interpolator includes means for 

2 shifting the off-grid position array to coincide with the pixel position array, whereby the 

3 resampled on-grid brightness value at each pixel position of the pixel position array is equal to 

4 the estimated off-grid brightness value at a coincident off-grid position of the shifted off-grid 

5 position array. 

1 141 . The apparatus of claim 138, wherein the interpolator derives a respective resampled 
.-=2 on-grid brightness value at a pixel position received from the second pixel position generator by 
yi3 computing the product of respective estimated off-grid brightness values at a predetermined 

4 number of off-grid positions that are nearest neighbors to the received pixel position and an 

':ll5 interpolation kernel having the same dimensions as the predetermined number of off-grid 
positions. 

Q 1 1 42. The apparatus of claim 141 , wherein the plurality of intermediate rows each extend 

2 medially between a respective adjacent pair of pixel rows and the plurality of regularly spaced 

3 intermediate columns each extend medially between a respective adjacent pair of pixel columns, 

4 and wherein the pixel positions provided by the second pixel position generator have coordinates 

5 (x,y) where x has values fi-om 2.5 to xSize-2.5 and, y has values from 2.5 to ySize-2.5, xSize is 

6 the dimension of the captured image in the direction of the pixel rows and ySize is the dimension 

7 of the captured image in the direction of the pixel columns. 
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1 143. The apparatus of claim 142, wherein the interpolator is responsive to a pixel position 

2 (x,y) received from the second pixel position generator for deriving a respective resampled on- 

3 grid brightness value in accordance with the relation 

/(x-2,^-2) - E i: 5(x-1.5 + Uy~h5 

1=0 J=0 ^ 

where I(x-2, y-2) is the resampled on-grid brightness value at pixel position (x-2, y-2), 

ijl5 B(x- 1 .5+i, y- 1 .5+j) is the estimated off^grid brightness value at off^grid position 

^"^6 (x- 1 .5+i, y- 1 .5+j), (j)y is the value of a 4 x 4 cubic interpolation kernel (j) at indices i and j, and 

|1^7 indices i and j each has values from 0 to 3. 

fyl 144. The system of claim 123, wherein the pixel positions provided by the second pixel 

^32 position generator are located at respective intersections of a plurality of regularly spaced pixel 

3 rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the 

4 pixel columns, and wherein the off-grid positions provided by the off-grid position generator are 

5 located at respective intersections of a plurality of regularly spaced intermediate rows and the 

6 pixel columns, and at respective intersections of a plurality of regularly spaced intermediate 

7 columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows 

8 and extending medially between a respective adjacent pair of the pixel rows, each intermediate 
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9 column being parallel to the pixel columns and extending medially between a respective adjacent 

10 pair of the pixel columns, the pixel positions provided by the second pixel position generator 

1 1 having coordinates (x,y) where x has values from 2.5 to xSize-2.5, y has values from 2.5 to 

12 y Size- 2.5, xSize is the dimension of the captured image in the direction of the pixel rows and 

13 ySize is the dimension of the captured image in the direction of the pixel columns. 

1 145. The system of claim 144, wherein the interpolator is responsive to a pixel position 

,.::=2 received from the second pixel position generator for deriving a respective resampled on-grid 

•! Ji3 brightness value in accordance with the relation 

£ I(x-2,y-2) - j2 E 5(x+0.5f-0.5/,>^-L5+0.5/^0.5/)(l) 

'^l^ i=o j=0 ^ 

[04 where I(x- 2,y- 2) is the resampled brightness value at pixel position (x- 2, y- 2), 

O5 B(x+0.5i-0.5j, y- 1 .5+0.5i+0.5j) is the off^grid brightness value at off^grid position 

6 (x+0.5i-0.5j, y- 1.5+0.5i+0.5j), ({)y is the value of a 4 x 4 cubic interpolation kernel (f) at indices i 

7 and j, and the indices i and j each have values from 0 to 3. 
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146. The method of claim 128 or 143, wherein the 4 x 4 cubic interpolation kernel (j) has 
the form: 
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147. The apparatus of claim 141^ wherein the interpolation kernel is a 2-dimensional 
Gaussian kernel. 

148. The system of claim 147, wherein the 2-dimensional Gaussian kernel has the form: 
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149. The system of claim 141, wherein the interpolation kernel is a bi-cubic B-spline 
interpolation filter kernel. 
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150. The system of claim 149, wherein the bi-cubic B-spline interpolation filter kernel 
has the form: 



0.0043 
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151. The apparatus of claim 141, wherein the interpolation kemel is a bi-linear 
interpolation filter kemel. 

152. The system of claim 151, wherein the bi-linear interpolation kemel has the form: 



0.5 


0.5 


0.5 


0.5 
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153. The system of claim 137, wherein the off-grid positions provided by the off-grid 
position generator are each located midway between a respective adjacent pair of pixel positions 
of the linear pixel position array, the pixel positions provided by the second pixel position 



NY02'241764.1 



- 165- 



FILE NO. A32708-070050.1 125 

4 generator having coordinate x, where x has values from 3.5 to xSize-3.5 and xSize is the 

5 dimension of the captured Une image. 

1 1 54. The system of claim 1 53, wherein the interpolator is responsive to a pixel position 

2 received from the second pixel position generator for deriving a respective resampled on-grid 

3 position brightness value in accordance with the relation 

^ k=-L5 

105 where Io(x-3) is the resampled on-grid brightness value at pixel position (x-3), B(x+k) is the 

6 estimated off-grid brightness value at off-grid position (x+k), k is a position of a 4 x 1 cubic 
i^yv interpolation kernel i|j(k) is the value of the cubic interpolation kernel ijf at position k, and the 
=38 position k has values from - L5 to 1.5. 

1 155. The system of claim 154, wherein the 4 x 1 cubic interpolation kemel i|; has the 

2 form: 
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1 1 56. A system for obtaining a relatively high dynamic range image of a scene using a 

2 relatively low dynamic range image sensor adapted to be exposed to incident light from the scene 

3 for capturing an image thereof comprising: 

4 (a) an image sensor having a multiplicity of light-sensing elements in an array, each one 

5 of the light-sensing elements having a particular one of a plurality of sensitivity levels to incident 

6 light in accordance with a predetermined sensitivity pattem for the array of light-sensing 
elements and having a response function, each one of the light-sensing elements in response to 

\M incident light from the scene producing a captured image brightness value at a corresponding one 

^^f 9 of a multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity 

'to of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light- 

r|l sensing elements; 

ri2 (b) a captured image memory for storing respective captured image brightness values 

^43 produced by the light-sensing elements at pixel positions of the pixel position array; 

14 (c) a pixel position generator for providing the pixel positions of the pixel position array; 

15 (d) an on-grid brightness value generator receiving the pixel positions from the pixel 

16 position generator and being responsive to a pixel position received from the pixel position 

17 generator for deriving from the captured image brightness values in the captured image memory 

18 a respective output image brightness value at the received pixel position, the respective output 
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19 image brightness value being derived from a corresponding plurality of captured image 

20 brightness values; and 

21 (e) an output image memory for storing the respective output image brightness value at 

22 each pixel position received from the pixel position generator derived by the on-grid brightness 

23 value generator. 

1 157. The system of claim 156, v^^herein the array of light-sensing elements of the image 

r-2 sensor is a two-dimensional array and the pixel position array is a linear array having regularly 

iJ13 spaced pixel positions in a linear pixel position row. 

;{:l 158. The system of claim 156, wherein the array of light-sensing elements of the image 

r-2 sensor is a two-dimensional array and the pixel position array is a two-dimensional array having 

\\B pixel positions located at respective intersections of a plurality of regularly spaced pixel rows and 

^34 a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel 

5 columns, and wherein the pixel positions provided by the pixel position generator have 

6 coordinates (x,y), where x has values from 2.5 to xSize-2.5, y has values from 2.5 to ySize-2.5, 

7 and xSize and ySize are the dimensions of the captured image in the directions of the pixel rows 

8 and the pixel columns, respectively. 
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1 1 59, The system of claim 158, wherein the on-grid brightness value generator is 

2 responsive to a pixel position (x,y) received from the pixel position generator for deriving from 

3 the captured image brightness values in the captured image memory a respective output image 

4 brightness value at the received pixel position in accordance v^ith the relation 

4 4 

I{x-l,y-2) = E E ^>-2 ^ Uy-2 

1=0 j-0 ^ 

where Ic(x - 2 + i, y - 2 + j) is the captured image brightness value at pixel position 

\jf> (x - 2 + i, y - 2 + j), is the value of an on-grid interpolation filter kernel (|) at indices i and j, 

^7 indices i and j each have values from 0 to 4, and Io(x - 2, y -2) is the output image brightness 

'^lJ8 value at pixel position (x - 2, y - 2). 

fyl 160. A system for obtaining a relatively high dynamic range image of a scene using a 

^31 relatively low dynamic range image sensor adapted to be exposed to incident light from the scene 

3 for capturing an image thereof comprising: 

4 (a) an image sensor having a multiplicity of light-sensing elements in an array, each one 

5 of the light-sensing elements having a particular one of a plurality of sensitivity levels to incident 

6 Ught in accordance with a predetermined sensitivity pattern for the array of light-sensing 

7 elements and having a response function, each light-sensing element in response to incident light 

8 from the scene producing a captured image brightness value at a corresponding one of a 
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9 multiplicity of pixel positions of a pixel position array, whereby each one of the mixltipUcity of 

1 0 pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light- 

1 1 sensing elements; 

12 (b) a captured image memory for storing the captured image brightness values produced 

1 3 by the light-sensing elements at pixel positions of the pixel position array; 

14 (c) a pixel position generator for providing the pixel positions of the pixel position array; 

1 5 (d) an on-grid brightness value generator receiving pixel positions from the pixel position 
..=4 6 generator and being responsive to a pixel position received from the pixel position generator for 
Lrt 7 deriving from the captured image brightness values in the captured image memory a respective 
M 8 on-grid brightness value at the received pixel position, the respective on-grid brightness value 

9 being derived from a corresponding plurality of captured image brightness values; 
i;§0 (e) a response fianction compensator for compensating the respective on-grid brightness 

f i 1 value at each received pixel position derived by the on-grid brightness value generator by the 

L32 inverse of a combined response function of the light-sensing elements to provide a respective 

23 output image brightness value at each pixel position received from the pixel position generator, 

24 the combined response function being the sum of weighted response functions of light-sensing 

25 elements having different ones of the plurality of sensitivity levels, each one of the weighted 

26 response functions being a response flmction of a light-sensing element having a respective one 

27 of the plurality of sensitivity levels multiplied by a predetermined weighting factor for the 

28 respective one of the plurality of sensitivity levels; and 
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29 (f) an output image memory for storing the respective output image brightness value at 

30 each received pixel positions provided by the response function compensator. 

1 161. The system of claim 1 60, wherein the response function compensator comprises a 

2 lookup table memory storing data representing the inverse of the combined response function of 

3 the light-sensing elements, and a mapper receiving on-grid brightness values from the on-grid 

4 brightness value generator and being responsive to an on-grid brightness value at a pixel position 
q5 received from the pixel position generator for deriving a respective index for the lookup table 
i:.n6 memory and providing a data value in the lookup table memory corresponding to the respective 

index, the data value being the output image brightness value at the pixel position received from 

;^g8 the pixel position generator. 

; U 1 1 62. The system of claim 1 60, wherein the array of light-sensing elements of the image 

::^2 sensor is a tv^o-dimensional array for capturing two-dimensional images, and the pixel position 

3 array is a two-dimensional array having pixel positions located at respective intersections of a 

4 plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the 

5 pixel rows being orthogonal to the pixel columns, and wherein the pixel positions provided by 

6 the pixel position generator having coordinates (x,y), where x has values from 2.5 to xSize-2.5, 

7 y has values from 2.5 to ySize-2.5, and xSize and ySize are the dimensions of the captured 

8 image in the directions of the pixel rows and the pixel columns, respectively. 
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1 163. The system of claim 162, wherein the on-grid brightness value generator is 

2 responsive to a pixel position received from the pixel position generator for deriving from the 

3 captured image brightness values in the captured image memory a respective on-grid brightness 

4 value at the received pixel position in accordance with the relation 

4 4 

I{x-2,y-2) = E E I{x~2 ^ Uy~2 ^j)^ 

i=o j=0 ^ 'J 

5 where I^Cx - 2 + i, y - 2 + j) is the captured image brightness value at pixel position 

6 (x - 2 + i, y - 2 + j), (f)ij is the value of a on-grid interpolation filter kernel c|) at indices i and j, the 

7 indices i and j each have values from 0 to 4, and I(x - 2, y -2) is the on-grid brightness value at 

8 pixel position (x - 2, y - 2). 
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164. The method of claim 159 or 163, wherein the on-grid interpolation filter kernel (j) 
has the form: 
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165. The system of claim 158, wherein the array of light-sensing elements of the image 
sensor is a linear array for capturing a line image, and the pixel position array is a linear array 
having regularly spaced pixel positions in a linear pixel row. 

166. A system for obtaining a relatively high dynamic range image of a scene using a 
relatively low dynamic range image sensor adapted to be exposed to incident light from the scene 
for capturing an image thereof comprising: 

(a) an image sensor having a multiplicity of light-sensing elements in an array, each one 
of the light-sensing elements having a particular one of a plurality of sensitivity levels to incident 
light in accordance with a predetermined sensitivity pattern for the array of light-sensing 
elements and having a respective response function, each light-sensing element in response to 
incident light from the scene producing a captured image brightness value at a corresponding one 
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9 of a multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity 

10 of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light- 

1 1 sensing elements; 

12 (b) a captured image memory for storing the captured image brightness values produced 

13 by the light-sensing elements at pixel positions of the pixel position array; 

14 (c) a first pixel position generator for providing the pixel positions of the pixel position 

15 array; 

i;|6 (d) a sensitivity pattern memory for storing data indicative of the sensitivity level 

li7 corresponding to each one of the first pixel positions of the pixel position array; 

8 (e) a response fimction compensator receiving pixel positions fi*om the first pixel position 

;^Pj9 generator, data indicative of the sensitivity levels corresponding to the received pixel positions 

r|0 fi-om the sensitivity pattern memory and captured image brightness values at the received pixel 

ill positions from the captured image memory, and being responsive to a pixel position received 

;:i2 from the first pixel position generator and data indicative of the sensitivity level corresponding to 

23 the received pixel position for retrieving from the captured image memory the captured image 

24 brightness value at the received pixel position and for compensating the retrieved captured image 

25 brightness value at the received pixel position by the inverse of the response fimction of a light- 

26 sensing element having the sensitivity level corresponding to the received pixel position to 

27 provide a respective compensated brightness value at the pixel position received from the first 

28 pixel position generator; 



NY02.241764.1 



- 174- 



FILE NO. A32708-070050.1125 



29 (f) a compensated on-grid brightness value memory for storing respective compensated 

30 captured image values at the pixel positions received from the first pixel position generator; 

31 (g) a second pixel position generator for providing the pixel positions of the pixel 

32 position array; 

33 (h) an on-grid brightness value generator receiving pixel positions from the second pixel 

34 position generator and being responsive to each pixel position received from the second pixel 

35 position generator for deriving from the captured image brightness values in the captured image 
i;|6 memory a respective output image brightness value at each received pixel position; and 

^ J7 (i) an output image memory for storing the respective output image brightness value at 

'|8 each pixel position received from the second pixel position generator. 

Cjl 1 67. The system of claim 1 66, wherein the response function compensator comprises a 

ry2 lookup table memory storing separate lookup table data representing the inverse of respective 

::;:3 response functions of light-sensing elements having different ones of the plurality of sensitivity 

4 levels, and a mapper receiving captured image brightness values from the captured image 

5 memory and being responsive to a captured image brightness value at a pixel position received 

6 from the first pixel position generator for deriving a respective index for the lookup table 

7 memory, and retrieving therefrom a compensated on-grid brightness value corresponding to the 

8 respective index from lookup table data representing the inverse of the response function of a 

9 light-sensing element having the sensitivity level corresponding to the received pixel position, 
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1 0 the retrieved compensated on-grid brightness value being provided to the compensated on-grid 

1 1 brightness value memory. 

1 168. The system of claim 166, wherein the array of light-sensing elements of the image 

2 sensor is a two-dimensional array, and the pixel position array is a two-dimensional array having 

3 pixel positions located at respective intersections of a plurality of regularly spaced pixel rows and 

4 a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel 
[jS columns. 

"^Z 1 1 69. The system of claim 1 66, wherein the array of light-sensing elements of the image 

■^g2 sensor is a linear array for capturing a line image and the pixel position array is a linear array 

f i3 having respective regularly spaced pixel positions of a linear pixel row, and wherein the pixel 

rU4 positions provided by the pixel position generator have coordinates x, where x has values from 

^:::::5 2.5 to xSize-2.5 and xSize is the dimension of the captured line image. 

1 170. The system of claim 169, wherein the on-grid brightness value generator is 

2 responsive to a pixel position x received from the second pixel position generator for deriving 

3 from the captured image brightness values in the captured image memory a respective output 

4 image brightness value at the received pixel position, the on-grid brightness value generator 

5 comprising: 
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6 (i) a memory for storing a 5 x 1 Gaussian interpolation filter kernel G; 

7 (ii) a comparator for comparing each one of five compensated on-grid brightness values 

8 I(k) at pixel positions k having values from x- 2 to x+2 with a predetermined low threshold value 

9 and a predetermined high threshold value; 

10 (iii) a multiplier for calculating a quantity I(k)G(k-x) for each value of k at which I(k) is 

1 1 greater than the predetermined low threshold value and less than the predetermined high 

12 threshold value, where G(k-x) is the value of the Gaussian interpolation kernel G at position 

:^i4 (iv) a first adder for adding the quantities I(k)G(k"x) calculated by the multiplier for each 

^15 value of k where I(k) is greater than the predetermined low threshold value and less than the 
I predetermined high threshold value to derive a first sum p; 

Cl7 (v) a second adder for adding the values G(k-x) of the Gaussian interpolation kernel in 

riS the memory for each value of k where I(k) is greater than the predetermined low threshold value 

]f 9 and less than the predetermined high threshold value to derive a second sum q; and 

20 (vi) a divider for deriving a respective output image brightness value Io(x) at the position 

21 X received fi:om the second pixel position generator by dividing the sum p by the sum q. 



171. The system of claim 1 70, wherein the predetermined low threshold value is the 
noise level of a light-sensing element having the sensitivity level corresponding to the pixel 
position of the compensated on-grid brightness value being compared therewith compensated by 
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4 the inverse of the response function of the light-sensing element, and the predetermined high 

5 threshold value is the saturation or near saturation brightness value of a light-sensing element 

6 having the sensitivity level corresponding to the pixel position of the compensated on-grid 

7 brightness value being compared therewith. 

1 172. The system of claim 170, wherein the 5 x 1 Gaussian interpolator filter kernel has 

2 the form: 
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ABSTRACT OF THE DISCLOSURE 
Disclosed are method and apparatus for obtaining relatively high dynamic range images 
using a relatively low dynamic range image sensor without significant loss of resolution. The 
image sensor has an array of light-sensing elements with different sensitivity levels in 
accordance with a predetermined spatially varying sensitivity pattern for the array of light- 
5 sensing elements. An image of a scene is captured with the image sensor and stored as 
Q brightness values at respective pixel positions in a linear or two-dimensional uniform grid. The 
in brightness values of the captured image at the pixel positions are then used to estimate the 

brightness values at off-grid positions of a uniform off-grid array located at respective interstices 
of the pixel position grid. The estimated off-grid brightness values are either used directly as the 
i;|0 pixel brightness values of a relatively high dynamic output image or interpolated to derive 
ry resampled on-grid brightness values at the pixel positions of the pixel position grid to provide a 

relatively high dynamic range output image. Alternatively, the brightness values of the captured 
image are interpolated by an on-grid interpolation filter to derive pixel brightness values of a 
relatively high dynamic range output image, each pixel brightness value of the output image 
1 5 being derived fi-om a corresponding plurality of the captured image brightness values. In each 
instance, either the captured image brightness values or the pixel brightness values of the output 
image may be compensated for non-linearities of the radiometric response function of the light- 
sensing elements of the image sensor. 



NY02-241764-1 



- 179- 



99-1 3-23: - 4:45 I 1«) VI-*W 



:0354486S06 # 




m 

m 




OCT 29 1999 32=00 



03544868K PflQE.32 



99-1 3-29; -4; <S : («) V;-:^W 



:0354436606 




3? 

(Q T3 
CD E. 



OCT 29 1959 02:01 



0354486806 PAGE 



99-1 0-29;- 4: iS : («) V--*W 



:0354436S06 



# 34 



Q 

» 

CO 











3 


c 


O 




a> 




CO 
0) 












Q 




ID 


< 


O 










Q 


CD 


O 


— ^ 





< = 
q: ! 

CD \ 




OCT 29 1999 02:01 



035448680S PAGE. 34 



99-1 3-29:: 4: ^£ : m^J^-m^ 



: 0354435606 



# 





:C3544366C6 



? 36 

I 




13 a a M s M A B a a n h si a 
°b"«SbSb?«^b*b*«%^b"b°b"b"b 

lBlB^BSB^BSi^«^.i?.%?B^B|B|B 
SBSBf«lBRB5i;?«ils!5Bf!B%i!B?B^B 

aniaaiiciBiiiB b^b^b^o b h 
^'^B'^iV/>!8^B^B^Bc^B>?fBv^fl^::B:^.B^9';^B^:B 

?!BS«SB!?B^«iBS^f?BfB??^B??.?BlB 
B&RPBSBBSBBfiia«B 

^■M;^B^^B;^^«^^>ai^;CI^^53^;ni-^IIt?a^^B*:^B^B?^B 

a o 9 B ih » m t» Tti B B fil d 

m n m m lY, ^ u m m rst m b b 

5^B%^B^«?«?^K;:^^E™;;i3t^J^^32r;;a:'i^B;;^B^^«^ 
^ B fis B ESI urn ^ ^ @ « B n 
^^^B^:^Bi^^^B^■^?;s^::^IB:;;a3■^2IIf^^m■;^v!S"^;^ 

K tS n B HI !J! ^ 3t r» B UJBJBJB^ 
^iBi^2B$^B^^B^=^si;j;Jffi:^EJi;0"^UI^a^;B£^B>-;!B^i^B 
19 B S B III [!E IS l£t !2 Q Q a 

i5iB?;?B^:l^^.^ W^^^nt'^?^ 
aBBBk£liinmi2Ei?t»3aBB 
v'^B:^a'^.:aL:B>:fi:^^IS:■^^3^•«^3S/>^!l^l>Br^^ 
aBBBBIKli^fU^SSj^lBaa 

^^B^^:B^;a^/'Bv-^B.':::BU8r»:^a^B^«>«^iB^i:B 

aSHaBBBBBBBBaPB 

^^i'B^B^»^B;^a-a;^:B^^B^B<^.^a'^a^B^B~c;B 

B B BB « a s BwaaaBBB 



OCT 29 1999 02=82 



035448e806 PAGE. 36 



: 0 5 5 4^,36806 



# 




□CT 29 1999 02:02 



0354486906 Pf=)GE, 37 



S9-i3-2S:-4:4S : myi-jM-m 



: 3354436806 



P 38 




OCT 29 1999 02=02 



0354486806 PAGE. 38 



3S-1 3-23; -4:49 : m'J-~mm 



: G3544366 C6 




OCT 29 1999 82=82 



035448680S PAGE 



3S-i3-2S:-4:4S : (^) V--«5SSl 



: 0;3 5 4 48 66 0 6 



P 4C 




OCT 29 1999 02:03 



0354486806 PAGE. 40 



9$-13-29:'4:iS 



; 03 54^3 56 0 6 




OCT 29 1999 02:03 



0354486806 PPGE.41 



99-1 0-23; -4 ;is : (?) v::-7ijF}lR 



: 33 54436606 



# 42 




^ >^ V 



OCT 29 1999 02:03 



0354486806 



PAGE, 42 




OCT 29 1999 02:03 



0354486806 PAGE. 43 



39-13-29: -4: 4S ; (fl) V--^iir>ffiT 



:0354435£06 



;t 44 




OCT 29 1999 02:03 



0354486806 PAGE. 44 



99-1 D-2S: -4:43 : («) V--*M 



0354486S06 



# 45 



Q 
in. 

m 

rU 

!=y 

iy 
□ 
□ 



I 



m 
z 




OCT 29 1999 02=03 



0:S4486806 PAGE. 45 



99-1 3-29: - 4:49 : V--*M 



: 03 544368 0 6 



# 46 



□ 
in 

cn 
□ 

ru 
ru 
w 



m 

O 



o 
c 



ro 



T 
r 

G 
H 



T 

o 
a 

3 



DO 
-1 



o 



OCT 29 1999 02:03 



0354486806 PAGE, 46 



99-1 0-23 I -4: 45 : («) Vl-^M 



: 03 544368 0 6 




5S-i3-2S:-4:4S ; (^) V--?:.W 



:Co54^S5c0e 




OCT 29 1999 02:03 



0354486806 PAGE. 48 



89-1 3 -29: - 4:49 : tS) VI-:^W 



:03 5U36S05 



# 52 




OCT 29 1999 02:84 



3354486806 PAGE. 52 



SS-n-23;-4:4S : {m'J--t%m^ 



:0354i86606 



# 52 



3 



m 

z: 
o 



T 

CO 







t 


T 




Co 
+ 


+ 






3^ 




r 
c 

t 

O 
cr 
o 
o 

r 

G 
H 



C3> 



o 
»-t 

o 
o 

Si 

T 

O 



M 
I 

O 



o 
n 

T 

o 
La 

o 

I 

O 





For 


For 




n 


each 




ich 








T 1 


T 


t 




o 


6 




( 


1 


CD 









CO 
H 



00 



OCT 29 1999 02:84 



0354486806 PAGE. 53 




29 1999 02:04 



0354486806 PAGE. 54 



93-1 3-23: - 4:49 : (mV--$&m 



:0354436c06 



# 43 





i 



OCT 29 1999 02:03 



0354486806 PAGE. 49 



0354^36606 



# 5C 




OCT 29 1999 02:03 



0354486806 PAGE , 50 



99-1 3-29; -4:49 : {m'J--tm^ 



:C3S44366C6 



# 51 



C3 

m 
+ 

Q 

m 
m 
w 



2 

i 



u) 

0 




OCT 29 1999 02:04 



0354485806 PAGE. 51 



9S-n-2S:'4:4S : (^) :G 354 ta6S06 # 66 




OCT 29 1999 92:85 



0354486806 PAGE, 66 



0354^35806 



# 55 





OCT 29 1999 32:04 



8354486806 PAGE. 56 



99-13-2S:-4:iS : (mV--rWm 



0354436606 



#t 57 




OCT 29 1999 02:04 



0354486806 PAGE. 57 



SS-1 3-29: -4:45 : {m'J-'^m?i 



: C554436SC6 



# 58 



TI 



^1 




OCT 29 1959 02:04 



0354486806 PAGE. 58 



99-1 3 -2 9: -4: 45 ; z-r.&%fi 



03 54:.3 68 0 6 




OCT 29 1999 02:04 



0354486806 PAGE. 59 



93-1 3-29; -4:43 I («) V--^M 



:0354436606 



# 6C 



4^ 



m 




OCT 29 1999 02:04 



0354486806 PAGE. 60 




OCT 23 1999 02:05 



0354486806 PAGE. 61 



99-1 3-29: - 4:45 ■.{m'J--mm 



# 62 




m 

D 



CO 
H 
> 















T 


For each 








1 












1 






t 


M 




1 

»— * 


to 


T 


«^ — 


LA 

+ 


Ln 
o 














1 








+ 


to 



















o 

fa 
O 

T 

to 
La 



to 



o 



OCT 29 1999 02:05 



0354486806 PPGE,62 



: C3 54:.3 5SG6 




OCT 29 1999 02:05 



0354486806 PAGE. 63 



: 0 3 54i;35£05 



# 65 



1^ 



m 
z 

D 



to 

I 

to 

t 



P3 

r 

a 



T 



to 
+ 



lO 

+ 



o 

O 



T 

to 
Ln 

Co 

^^ 
I 

u 



CD 

n 

T 

to 

B 

X 
Co 

to 



H 
> 



I 



OCT 29 1999 02:25 



0354486606 PAGE. 65 



93-1 3-25: - 4:43 : «) VI-«?T 



:!1354iS5SG6 



# 64 



a 
m 

m 

a 
m 
ru 

□ 

£3 



3 

1 



0.043 


-0.618 


-1.322 


-0.618 


0.043 


-0.618 


9.001 


19.238 


9.001 


-0.618 


-1 .322 


19.238 


41.12 


19.238 


-1.322 


-0.618 


9.001 


19.238 


9.001 


-0.618 


0.043 


-0.618 


-1.322 


-0.618 


0.043 



t 

OCT 29 1999 82:05 



0354486886 PAGE. 64 



277 



scene 




3 ' 



\ 

J) 7^ 



motion 



Figure 



so 





motion conveyor belt 

Figure €M 



^/i 2/^ z6i -2'^ 2(J? J 2, 



Figure'# ^ 




Time 



^5 

Figure 



On-Grid Off-Grid 

r -r JU^ 



Figure'^ 



01/05/00 00:04 FAX 



■a 



Start 



For Each 0.5 to xSize-0.5 



LUTj-^ ChooseLUT(S(x)) 



I 



r 



Wx) LUTi(Ie(x)) 



r 



l91 



For Each x'-^ 2 to xSize -2 









q-i-0 






1 







For Each k^x'-1.5 to x'+1.5 

7^4 




p = p + l„(k) G(k-x') 

q = q + G(k-x') 



jjf ^Continue^ 



B(x')-»-p/q 



Zf3 



End 



JAN 04 2000 12^04 



PPGE.03 




0.1 


3.6 


10.0 


3.6 


0.1 



61^ 



01/10/00 23:06 FAX 



12102 



-0.96 


13.44 


13.44 


-0.96 




f^f Si 

ginn iTnrnpl ( feiT ni 



Cn 

''4 



Start 



1 


r 




For Each x 3.5 to xSize -3.5 




k = 1.5 

Io('^-3)^X B(x + k)i}/(k) 
k = -1.5 










\ 


f 



End 



2^; 




JfiN 10 2000 li:05 



PPGE.02 



01/10/00 23:06 FAI 



Sensitivity 
Pattern 



20 



Captured 
Image 



V2 



Mapper 



¥7 




LUT 




mapped 
brightness 



3^ 



SIM. 




2i 



Output 
Image 



Zm 10 2000 11 = 06 



Fii III! - niiiwBBii 



PAGE. 03 



start 



For Each X-*— 0.5 to xSize-0.5 



LUTj<- ChooseLUT(S(x)) 



Ux) LUTj (I, (X)) 



For Each x -*-2.5 to xSize -2.5 

30^ 





0 




0 



30 £ 



ForEachk^x-2 to x+2 -j 




P = P + ln(k) G(k-x) 
q = q + G(k-x) 



^Continue) 



Io(x>»-p/q 



30^ 



End 




JPIN 04 2000 12:04 



PAGE. 02 



BAKER BOTTS L.L.R 

^ ' FILE NO.: A32708-070050.1125 

COMBINED DECLARATION 
AND POWER OF ATTORNEY 

(Original, Design, National Stage of PCT, Divisional, Continuation or C-I-P Application) 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name; I believe I am the original, 
first and sole inventor (if only one name is listed below) or an original, first and joint inventor (if plural names are 
listed below) of the subject matter which is claimed and for which a patent is sought on the invention entitled: 

METHOD AND APPARATUS FOR OBTAINING HIGH DYNAMIC RANGE IMAGES 

This declaration is of the following type: 

[X] original 
[] design 

[] national stage of PCT, 
[] divisional 
[] continuation 
[] continuation-in-part (C-I-P) 

[Sthe specification of which: (complete (a), (b), or (c)) 

i-^a) [X ] is attached hereto. 

;P(b) [ ] was filed on as Application Serial No. and was amended on (if applicable), 

-Jj(c) [ ] was described and claimed in PCT International Application No. filed on and was amended on (if 

"'^'applicable). 

i y Acknowledgement of Review of Papers and Duty of Candor 

m I hereby state that I have reviewed and understand the contents of the above identified specification, 
:;;jncluding the claims, as amended by any amendment referred to above. 

Q I acknowledge the duty to disclose information which is material to the patentability of the subject matter 

^;;3claimed in this application in accordance with Title 37, Code of Federal Regulations § 1.56. 

[ ] In compliance with this duty there is attached an information disclosure statement. 37 CFR 1.98. 

Priority Claim 

I hereby claim foreign priority benefits under Title 35, United States Code, § 119(a)-(d) of any foreign 
application(s) for patent or inventof s certificate or of any PCT International Application(s) designating at least one 
country other than the United States of America listed below and have also identified below any foreign 
application(s) for patent or inventor's certificate or any PCT International Apphcation(s) designating at least one 
country other than the United States of America filed by me on the same subject matter having a filing date before 
that of the application on which priority is claimed 



(complete (d) or (e)) 

(d) [X ] no such applications have been filed. 

(e) [ ] such applications have been filed as follows: 



BAKER BOTTS L.L.P. 

FILE NO.: A32708-070050.1125 



PRIOR FOREIGN/PCT APPUCATION(S) FILED WITHIN 12 MONTHS (6 MONTHS FOR DESIGN) PRIOR TO SAID APPLICATION 


COUNTRY APPLICATION NO 


DATE OF FILING 
(day, month, year) 


DATE OF ISSUE 
(day, month, year) 


PRIORITY CLAIMED 
UNDER 35 use 119 








[ ] YES NO [ ] 








[ ] YES NO [ ] 








[ ] YES NO [ ] 


ALL FOREIGN APPL1CATI0N[S], IF ANY, FILED MORE THAN 12 MONTHS (6 MONTHS FOR DESIGN) PRIOR TO SAID APPLICATION 










[ ] YES NO [ ] 








[ ] YES NO [ ] 








[ ] YES NO [ ] 



Claim for Benefit of Prior U.S. Provisional Application(s) 

I hereby claim the benefit under Title 35, United States Code, § 119(e) of any United States provisional 
appUcation(s) listed below: 



Provisional Application Number 


Filmg Date 















5 Claim for Benefit of Earlier U.S./PCT Application(s) under 35 U.S.C. 1 20 

1 ( complete this part only if this is a divisional, continuation or C-I-P application ) 
, I hereby claim the benefit under Title 35, United States Code, § 120 of any United States application(s) or 

^PCT mtemational application(s) designating the United States of America that is/are listed below and, insofar as 
=the subject matter of each of the claims of this application is not disclosed in the prior application(s) in the manner 
^provided by the first paragraph of Title 35, United States Code § 112, I acknowledge the duty to disclose 
^mformation as defined in Title 37, Code of Federal Regulations, § 1.56 which occurred between the filing date of 
,the pnor application(s) and the national or PCT international filing date of this application: 



!.r! (Application Serial No ) 



(Filing Date) 



(Status) (patented, pending, abandoned) 



(Status) (patented, pending, abandoned) 



^(Application Serial No ) (Frlmy Date) 

Power of Attorney 

As a named inventor, I hereby appoint Dana M.Raymond, Reg. No. 1 8,540; Frederick C. Carver, Reg. No 17 021- Francis J Hone Res 
No. 18,662; Joseph D. Garon, Reg. No. 20,420; Arthur S. Tenser, Reg. No. 18,839; Ronald B. Hildreth, Reg. No. 19 498- Thomas R 
Nesbitt, Jr., Reg. No. 22,075; Robert Neuner, Reg. No. 24,3 16; Richard G. Berkley, Reg. No. 25,465; Richard S. Clark Reg No 26 154- 
Bradley BGeist, Reg. No. 27,551; James J. Maune, Reg. No. 26,946; John D. Mumane, Reg. No. 29,836; Henry Tang, Reg' No' 29705' 
Robert C Schemfeld, Reg. No. 31,300; John A. Fogarty, Jr., Reg. No. 22,348; Louis S. Sorell, Reg. No. 32,43^ Rochelle K. SeideReg 
BAKERBO???'^' '^kT'^tl: E^pelsignore, Reg. No. 32,689; and Lisa B. Kole, Reg. No. 35,225 of the firm of 

BAKER BOTTS L.L.P with offices at 30 Rockefeller Plaza, New York, New York 1 0 1 12, as attorneys to prosecute this application and 
to transact all business m the Patent and Trademark Office connected dierewith 



SEND CORRESPONDENCE TO: 
BAKER BOTTS L.L.P. 

30 ROCKEFELLER PLAZA, NEW YORK, N.Y 101 12 
CUSTOMER NUMBER: 21003 



DIRECT TELEPHONE CALLS TO: 

BAKER BOTTS L.L.P. 
(212) 705-5000 



I hereby declare that all statements made herem of my own knowledge are true and that all statements made 
on mformation and belief are beheved to be true; and further that these statements were made with the knowledge 



-2- 



BAKER BOTTSL.L.P. 

FILE NO.: A32708-070050.1125 
that willful false statements and the like so made are punishable by fine or imprisonment, or both, under Section 
1001 of Title 1 8 of the United States Code and that such willful false statements may jeopardize the vahdity of the 
application or any patent issued thereon. 



FULL NAME OF SOLE 
OR FIRST INVENTOR 


LAST NAME 

Nayar 


FIRST NAME 

Shree 


MIDDLE NAME 

K. 


RESIDENCE & CITIZENSHIP 


CITY 

New York 


STATE or FOREIGN COUNTRY 

New York 


COUNTRY OF CITIZENSHIP 

US. 


POST OFFICE 
ADDRESS 


POST OFFICE ADDRESS 

560 Riverside Drive 


CITY 

New York , 


STATE or COUNTRY 

New York 


ZIP CODE 

10027 


DATE . 1 
0^1 lojoo 


SIGNATURE OF INVENTOR j 

I 




FULL NAME OF SECOND 
JOINT INVENTOR, IF ANY 


LAST NAME 

Mitsunaga 


FIRST NAME 

Tomoo 


MiDDLelslAME 


RESIDENCE & CITIZENSHIP 


CITY 

Kawasaki City 


STATE or FOREIGN COUNTRY 

Japan 


COUNTRY OF CITIZENSHIP 

Japan 


POST OFFICE 
ADDRESS 


POST OFFICE ADDRESS 

1-9-3-403, Hisamoto, Takatsu 


CITY 

Kawasaki City, 
Kanagawa Prefecture 


STATE or COUNTRY 

Japan 


ZIP CODE 

213-0011 


DATE 


SIGNATURE OF INVENTOR 




FULL NAME OF THIRD 
JOINT INVENTOR, IF ANY 


LAST NAME 


FIRST NAME 


MIDDLE NAME 


RESIDENCE & CITIZENSHIP 


CITY 


STATE or FOREIGN COUNTRY 


COUNTRY OF CITIZENSHIP 


POST OFFICE 
ADDRESS 


POST OFFICE ADDRESS 


CITY 


STATE or COUNTRY 


ZIP CODE 


DATE 


SIGNATURE OF INVENTOR 




FULL NAME OF FOURTH 
JOINT INVENTOR, IF ANY 


LAST NAME 


FIRST NAME 


MIDDLE NAME 


RESIDENCE & CITIZENSHIP 


CITY 


STATE or FOREIGN COUNTRY 


COUNTRY OF CITIZENSHIP 


POST OFFICE 
ADDRESS 


POST OFFICE ADDRESS 


CITY 


STATE or COUNTRY 


ZIP CODE 


DATE 


SIGNATURE OF INVENTOR 




FULL NAME OF FIFTH 
JOINT INVENTOR, IF ANY 


LAST NAME 


FIRST NAME 


MIDDLE NAME 


RESIDENCE & CITIZENSHIP 


CITY 


STATE or FOREIGN COUNTRY 


COUNTRY OF CITIZENSHIP 


POST OFFICE 
ADDRESS 


POST OFFICE ADDRESS 


CITY 


STATE or COUNTRY 


ZIP CODE 


DATE 


SIGNATURE OF INVENTOR 






FULL NAME OF SIXTH 
JOINT INVENTOR. IF ANY 


LAST NAME 


FIRST NAME 


MIDDLE NAME 


RESIDENCE & CITIZENSHIP 


CITY 


STATE or FOREIGN COUNTRY 


COUNTRY OF CITIZENSHIP 


POST OFFICE 
ADDRESS 


POST OFFICE ADDRESS 


CITY 


STATE or COUNTRY 


ZIP CODE 


DATE 


SIGNATURE OF INVENTOR 
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FULL NAME OF SEVENTH 

r^IXIT IX1\/PXIT/^P TP A\TV 
JVJilNI 1 IN VtfN i \JK, Ir AIN I 


LAST NAME 


FIRST NAME 


MIDDLE NAME 


RESIDENCE & CITIZENSHIP 


CITY 


STATE or FOREIGN COUNTRY 


COUNTRY OF CITIZENSHIP 


POST OFFICE 
ADDRESS' 


POST OFFICE ADDRESS 


CITY 


STATE or COUNTRY 


ZIP CODE 



Check proper boxfes) for any added page(s) forming a part of this declaration 

[ ] Signature for ninth and subsequent joint inventors. Number of pages added . 

[ ] Signature by administrator(trix), executor(trix) or legal representative for deceased or mcapacitated inventor. 

Number of pages added 

[ ] Signature for inventor who refuses to sign, or cannot be reached, by person authorized imder 37 CFR 1.47. 
Number of pages added . 
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COMBINED DECLARATION 
AND POWER OF ATTORNEY 

(Original, Design, National Stage of PCT, Divisional, Continuation or C-I-P Application) 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name; I beUeve I am the original, 
first and sole inventor (if only one name is listed below) or an original, first and joint inventor (if plural names are 
listed below) of the subject matter which is claimed and for which a patent is sought on the invention entitled: 

METHOD AND APPARATUS FOR OBTAINING HIGH DYNAMIC RANGE IMAGES 

This declaration is of the following type: 

[X] original 
[] design 

[] national stage of PCT. 
[] divisional 
[] continuation 
[] continuation-in-part (C-I-P) 

i n the specification of which: (complete (a), (b), or (c)) 

M (a) [X ] is attached hereto. 

£(b) [ ] was filed on as Application Serial No. and was amended on 
■^L:(c) [] was described and claimed in PCT International Application No 

"'■^applicable). 

rij Acknowledgement of Review of Papers and Duty of Candor 

m I hereby state that I have reviewed and understand the contents of the above identified specification, 

i^y including the claims, as amended by any amendment referred to above. 

Cj I acknowledge the duty to disclose information which is material to the patentability of the subject matter 
□claimed in this application in accordance with Title 37, Code of Federal Regulations § 1.56. 

[ ] In compliance with this duty there is attached an information disclosure statement. 37 CFR 1.98. 

Priority Claim 

I hereby claim foreign priority benefits under Title 35, United States Code, § 119(a)-(d) of any foreign 
application(s) for patent or inventor^s certificate or of any PCT International Application(s) designating at least one 
country other than the United States of America listed below and have also identified below any foreign 
application(s) for patent or inventor's certificate or any PCT International Application(s) designating at least one 
country other than the United States of America filed by me on the same subject matter having a filing date before 
that of the application on which priority is claimed 

(complete (d) or (e)) 

(d) [X ] no such appUcations have been filed. 

(e) [ ] such apphcations have been filed as follows: 




(if applicable). 
filed on and was amended on (if 
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PRIOR FOREIGN/PCX APPLICATION(S) FILED WITHIN 12 W NTHS (6 MONTHS FOR DESIGN) PRIOR TO SAID APPLICATIO> 


COUNTRY APPLICATION NO. 


DATE OF HLING 
(day, month, year) 


DATE OFISn.V, 
(day, month. - . 


PRIORITY CLAIMED 
UNDER 35 use 119 








[ ] YES NO [ ] 








[ ] YES NO [ ] 








[ ] YES NO [ ] 


ALL FOREIGN APPLICATION[S], IF ANY, FILED MORE THAN 12 MONTHS (6 MONTHS FOR DESIGN) PRIOR TO SAID APPLICATION 








[ ] YES NO [ ] 








[ ] YES NO [ ] 








[ ] YES NO [ ] 



Claim for Benefit of Prior U.S. Provisional AppIication($) 

I hereby claim the benefit under Title 35, United States Code, § 11 9(e) of any United States provisional 
application(s) listed below; 



Provisional Application Number 


Filing Date 















5 Claim for Benefit of Earlier U.S./PCT Application(s) under 35 U.S.C. 120 

Ji (complete this part only if this is a divisional, continuation or C-LP application) 

I hereby claim the benefit under Title 35, United States Code, § 120 of any United States application(s) or 
=y CT international application(s) designating the United States of America that is/are listed below and, insofar as 
;hhe subject matter of each of the claims of this appHcation is not disclosed in the prior application(s) in the manner 
=J,provided by the first paragraph of Title 35, United States Code § 112, I acknowledge the duty to disclose 
'^Information as defined in Title 37, Code of Federal Regulations, § 1.56 which occurred between the filing date of 
-jthe prior apphcation(s) and the national or PCT international filing date of this application: 



rf( Application Serial No ) 



(Filing Date) 



(Status) (patented, pending, abandoned) 



^(Application Senal No.) (Fiimg Date) (Status) (patented, pending, abandoned) 

Power of Attorney 

As a named inventor, I hereby appoint Dana M. Raymond, Reg. No. 1 8,540; Frederick C. Carver, Reg. No. 1 7,02 1 ; Francis I Hone, Reg. 
No. 18,662; Joseph D. Garon, Reg. No. 20,420; Arthur S. Tenser, Reg. No. 18,839; Ronald B. Hildreth, Reg. No. 19,498; Thomas R. 
Nesbitt, Jr., Reg. No. 22,075; Robert Neuner, Reg. No. 24,316; Richard G. Berkley, Reg. No. 25,465; Richard S. Clark, Reg. No. 26,154; 
Bradley B. Geist, Reg. No. 27,55 1 ; James J. Maune, Reg. No, 26,946; John D. Mumane, Reg. No. 29,836; Henry Tang^ Reg. No. 291705; 
Robert C. Scheinfeld, Reg. No. 3 1,300; John A. Fogarty, Jr., Reg. No. 22,348; Louis S. Sorell, Reg. No. 32,439; Rochelle K. Seide Reg! 
No. 32,300; Gary M. Butter, Reg. No. 33,841 ; Marta E. Delsignore, Reg. No. 32,689; and Lisa B. Kole, Reg. No. 35,225 of the firni of 
BAKER BOTTS L.L.P., with offices at 30 Rockefeller Plaza, New York, New York 1 01 12, as attorneys to prosecute this application and 
to transact all business in the Patent and Trademark Office connected therewith 



SEND CORRESPONDENCE TO: 
BAKER BOTTS L.L.P. 

30 ROCKEFELLER PLAZA, NEW YORK, N. Y. 10112 
CUSTOMER NUMBER: 21003 



DIRECT TELEPHONE CALLS TO: 
BAKER BOTTS L.L.P. 
(212) 705-5000 



I hereby declare that all statements made herein of my own knowledge are true and that all statements made 
on information and belief are believed to be true; and further that these statements were made with the knowledge 
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that willful false statements and the like so made are punishable by fine or imprisonment, or both, under Section 
1 001 of Title 1 8 of the United States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issued thereon. 



FULL NAME OF SOLE 
OR FIRST INVENTOR 


LAST NAME 

Nayar 


FIRST NAME 

Shree 


MIDDLE NAME 

K. 


RESIDENCE & CITIZENSHIP 


CITY 

New York 


STATE or FOREIGN COUNTRY 

New York 


COUNTRY OF CITIZENSHIP 

U.S. 


POST OFFICE 

AUUKEii 


POST OFFICE ADDRESS 

560 Riverside Drive 


CITY 

New York 


STATE or COUNTRY 

New York 


ZIP CODE 

10027 


DATE 


SIGNATURE OF INVENTOR 




FULL NAME OF SECOND 
JOINT INVENTOR, IF ANY 


LAST NAME 

Mitsunaga 


FIRST NAME 

Tomoo 


MIDDLE NAME 


RESIDENCE & CITIZENSHIP 


CITY 

Kawasaki City 


STATE or FOREIGN COUNTRY 

Japan 


COUNTRY OF CITIZENSHIP 

Japan 


POST OFFICE 
ADDRESS 


POST OFFICE ADDRESS 

1-9-3-403, Hisamoto, Takatsu 


CITY 

Kawasaki City, 
Kanagawa Prefecture 


STATE or COUNTRY 

Japan 


ZIP CODE 

213-0011 


DATE 


SIGNATURE OF INVENTOR y 




FULL NAME OF THIRD 
JOINT INVENTOR, IF ANY 


LAST NAME 


FIRST NAME ^ 


MIDDLE NAME 


RESIDENCE & CITIZENSHIP 


CITY 


STATE or FOREIGN COUNTRY 


COUNTRY OF CITIZENSHIP 


POST OFFICE 
ADDRESS 


POST OFFICE ADDRESS 


CITY 


STATE or COUNTRY 


ZIP CODE 


DATE 


SIGNATURE OF INVENTOR 




FULL NAME OF FOURTH 
JOINT INVENTOR, IF ANY 


LAS I' NAME 


FIRST NAME 


MIDDLE NAME 


RESIDENCE & CITIZENSHIP 


CITY 


STATE or FOREIGN COUNTRY 


COUNTRY OF CITIZENSHIP 


POST OFFICE 
ADDRESS 


POST OFFICE ADDRESS 


CITY 


STATE or COUNTRY 


ZIP CODE 


DATE 


SIGNATURE OF INVENTOR 




FULL NAME OF FIFTH 
JOINT INVENTOR, IF ANY 


LAST NAME 


FIRST NAME 


MIDDLE NAME 


RESIDENCE & CITIZENSHIP 


CITY 


STATE or FOREIGN COUNTRY 


COUNTRY OF CITIZENSHIP 


POST OFFICE 

AUUKLoa 


POST OFFICE ADDRESS 


CITY 


STATE or COUNTRY 


ZIP CODE 


DATE 


SIGNATURE OF INVENTOR 






FULL NAME OF SIXTH 
JOINT INVENTOR, IF ANY 


LAST NAME 


FIRST NAME 


MIDDLE NAME 


RESIDENCE & CITIZENSHIP 


CITY 


STATE or FOREIGN COUNTRY 


COUNTRY OF CITIZENSHIP 


POST OFFICE 
ADDRESS 


POST OFFICE ADDRESS 


CITY 


STATE or COUNTRY 


ZIP CODE 


DATE 


SIGNATURE OF INVENTOR 
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FULL NAME OF SEVENTH 
JOINT INVENTOR, IF ANY 


LAST NAME 


FIRST NAME 


MIDDLE NAME 


RESIDENCE & CITIZENSHIP 


CITY 


STATE or FOREIGN COUNTRY 


COUNTRY OF CITIZENSHIP 


POST OFFICE 
ADDRESS' 


POST OFFICE ADDRESS 


CITY 


STATE or COUNTRY 


ZIP CODE 


DATE 


SIGNATURE OF INVENTOR 







Check proper box(e s) for any added page(s) forming a part of this declaration 

[] Signature fornmdi and subsequent joint inventors. Number of pages added . 

[ ] Signature by administrator(tnx), executor(trix) or legal representative for deceased or incapacitated inventor. 
Number of pages added ^ 



[ ] Signature for inventor who refuses to sign, or cannot be reached, by person authorized under 37 CFR 1.47. 
Number of pages added . 



